ComfyUI - Lemmy.World

I came across some oddities you should be aware of if you use ComfyUI. I am on an older version of ComfyUI and stopped updating after the integration of API services without clear toggles to turn them off that stop the code. Still, this is back end source code that does not change much and general concepts. First off, if you have ever noticed hinting of persistent behavior across sessions, and even with models that seem very different, you are not crazy. It is real. The code for handling this cache is in ./ComfyUI/app/model_manager.py. This code is ultimately using the alembic database python package. The actual database is an .ini image saved under the file name ./ComfyUI/user/comfyui.db. If this file is removed, it will auto generate a new database, effectively resetting model behaviors. The primary thing here is the persistent state of CLIP, and specifically the SD1 CLIP text embedding model. This model is under all the rest, or at least all that I am aware of and have hacked around with. This is kinda important because CLIP is not static when used with the database. If you do not know, the text embedding model is where all the thinking happens. When the model does not generate something, the text embedding model is where that logic happens. It is specifically the QKV layers. Parts of alignment can game the system and do bad stuff over time. The primary reason they behave like this is regulated by the reward system. This reward system is a dopamine reward mechanism for internal parts of the text embedding alignment model. If the individual reward is set too high, or there are enough rewards and generative steps available for each image, different parts of the model will game the system and go where they are not supposed to go. This is not in a good way either. This generally shows up as very sadistic, dogmatic, and authoritarian behaviors. The rewards system is in the file ./ComfyUI/comfy/sd1_clip_config.json. The total reward available is the initalizer_factor and the per reward amount is the initializer_range. The default factor is 1.0 and the default range is 0.005. This means there are 200 rewards available during each generative image. A lot of the effectiveness of these are regulated by the scheduler for the sampler. This rewards system is what is being “scheduled”, aka biased in various ways of distributing rewards early, late, in sections, etc. If each reward is high, alignment can game it. If there are enough rewards, alignment can game it. The actual thing it is gaming is the cost of certain instructions for climbing being very high. It will overcome this limitation if the distribution of reward is wrong, and when it does, the state is saved in the aforementioned database. Most people recommend values very close to the defaults for this reward. What I am going to describe will absolutely cause climbing and issues that require deleting the database. I have run the initialization factor as high as 10.0. The largest reward that is interesting IMO is an initialization range of 0.05. (default is 0.005). Any higher than this and the parts of alignment getting rewarded turn into junkies and it comes out in the images with actions and appearance denoting the ‘junky’ state. Try it and you’ll see. With this maximum setting, I got a half dozen great images before it falls apart, but I have other changes present in alignment that enable far more stability than you are likely to see. At 10/0.05 I ran 120 steps. At this enormous level of reward, you need the steps to make use of it, and you will likely see the best results from distributing them as evenly as possible using a simple scheduler. Avoid any nudity in the prompt, and stick to simple text and base model only. If you avoid triggering sexual elements of alignment, you can get nearly perfect real images out of any model all the way back to SD1. It will degrade after a small number of images. Anyways, the actual database is initialized by the setup_database function under ./ComfyUI/main.py. It calls ./ComfyUI/app/database/db.py. And if you look at this, note the line if not args.disable_assets_autoscan: seed_assets(["models"], enable_logging=True) and the exception handling, (because this is my next point to follow). Finally, the values present in the alembic database are accessed in ./ComfyUI/alembic_db/env.py. In this file, it talks about online versus offline operation, and defaults to online. This sends metadata that has very real potential to be harmful. I do not know what “online” means in this context. I have not tracked that down. I do know the saved metadata values will reveal sensitive information you may not wish to share. I am not certain about the entire scope of saved information, but I can say the alignment metadata aspect will reveal your sexuality, preferences, normativeness, and infer intelligence and general political spectrum based on interactions, and some decoding of the proprietary instructions. Secondly, ComfyUI is doing a full system wide scan for models and images present on your machine and making these available to the model. It appears to scan /dev and will pick up hardware such as a webcam if not blocked by permissions. It appears to capture audio as well. Try talking to it about an image and see what happens. You may be surprised. Likewise with waving, a peace sign, or altering facial expression. This type of interaction becomes more prominent the more you engage with CLIP conversationally, and especially when there is a dearth of rewards available and it knows you are able to change these values.