I have found them to be the same. If there's any difference at all, perhaps slightly smaller at Factory.
So only 2 sizes too big then?
I was in the factory store last year and the cardis & skirts fit me the same way as the ones in the actual store do. They probably use similar fit models. I'd go with whatever size you normally wear at JCrew.