Ideally, you should have unwrapped your UVs and baked the objects before placing them in the scene. Now, it still should be possible, but somewhat difficult. How you approach the task will depend on how your scene is setup.
Are the thousands of objects identical (or with a few variations)? Are they spread into the scene with particle systems? Then it would be easy, as you would need to unwrap and bake only the base objects for all the others to be updated.
Are they identical, but separate, individual objects? Then you unwrap and bake only one of them and use “link object data” to make them all share the same mesh and turn them into instances.
Are the objects all unique? I would bake everything into one material. For previs purposes, the automatic UV unwrap should be good enough for this task. Here is a tutorial explaining what I mean.
If you do it all correctly, you will have all the objects share a single material and a single set of textures.
If you have many objects, Blender will bake them one by one, so you will want to join everything that can be joined before baking, even if just temporarily. Also, you will probably need a huge texture (at least 8192x8192), and set the render samples to something really low for the bake (depending on the shaders, 1 sample may even be enough). This will take a while to bake, I am aware, but I don’t know a better alternative.
For the file export, I can tell you in advance that obj doesn’t do animation. FBX can, and will even export armatures (though you may need to play with the settings quite a bit).
A format I would look at is Alembic. It will remove all constraints and armatures and just export the final result of the geometry as a mesh sequence. So easy transfer of animated scenes between software, but then the rigs and constraints are gone.
