Why were the "good figs" punished?

God ways are mysterious.  It was like that back in the day and it is like that now as well.  This verse is a great example.  Who would have thought that those who were being sent into captivity were actually receiving a blessing because those who were left behind were left behind to die a terrible death. 

Jeremiah 24:4-7(NKJV)

4Again the word of the Lord came to me, saying, 

5“Thus says the Lord, the God of Israel: ‘Like these good figs, so will I acknowledge those who are carried away captive from Judah, whom I have sent out of this place for their own good, into the land of the Chaldeans. 

6For I will set My eyes on them for good, and I will bring them back to this land; I will build them and not pull them down, and I will plant them and not pluck them up. 

7Then I will give them a heart to know Me, that I am the Lord; and they shall be My people, and I will be their God, for they shall return to Me with their whole heart.

What I find interesting here is that even the "good figs" were punished by being taken into captivity.  Their punishment wasn't nearly as severe as the "bad figs" but they were punished just the same.  Scripture says that they were sent into captivity for their own good but it seems to me that is was still a punishment.

So the question becomes, why were the "good figs" punished if they were good?  My guess is that they bore some guilt for not standing more aggressively against evil but that is just a guess on my part.