Quick Trick to anyone who had the same dilemma as me.
How to get value from collection with one-to-many relationship ?
The solution is really simple. Use Dot notation !
Post::with('users:id')->get()->pluck('users.*.id')->collapse();
Simple and elegant.