ai/ai
ai/ai
1
0
Fork 0
Code Packages

first

Browse Source
This commit is contained in:
syui 2025-03-24 14:13:40 +09:00
commit 0ac87a0c25
Signed by: syui
GPG Key ID: 5417CFEBAD92DF56
7 changed files with 1738 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
README.md Normal file
View File

@ -0,0 +1,16 @@
# <img src="./icon/ai.png" width="30"> ai `ai`
AI model `ai`
the aim is to incorporate it into `aibot` and `aios`.
|name|full|code|repo|
|---|---|---|---|
|ai|ai ai|aiai|https://git.syui.ai/ai/ai|
|os|ai os|aios|https://git.syui.ai/ai/os|
|bot|ai bot|aibot|https://git.syui.ai/ai/bot|
|at|ai|ai.syu.is|https://git.syui.ai/ai/at|
```sh
$ ollama run syui/ai "hello"
```

329
docs/ja.md Normal file
View File

@ -0,0 +1,329 @@
# ai
AI modelの`ai`は、`aibot`や`aios`に組み込むことを目指します。
|name|full|code|repo|
|---|---|---|---|
|ai|ai ai|aiai|https://git.syui.ai/ai/ai|
|os|ai os|aios|https://git.syui.ai/ai/os|
|bot|ai bot|aibot|https://git.syui.ai/ai/bot|
|at|ai|ai.syu.is|https://git.syui.ai/ai/at|
```sh
$ ollama run syui/ai "hello"
```
## 学習
物語を学習させます。特定の語彙を使用します。例えば「自分のことをアイという」などです。
> アイね、回答するの
## できること
基本的には`aibot`へのrequestに応じて、`comfyui`で画像や動画生成、LLMで回答を行います。
webからはatprotoを通じて実行されます。
```sh
[web]aiat --> [server]aios --> [at]aibot --> [ai]aiai
```
## 使用するもの
- https://github.com/ollama/ollama
- https://github.com/n8n-io/n8n
- https://github.com/comfyanonymous/comfyui
- https://github.com/NVIDIA/cosmos
- https://github.com/stability-ai/stablediffusion
- https://github.com/unslothai/unsloth
- https://github.com/ml-explore/mlx-examples
- https://github.com/ggml-org/llama.cpp
```json
{
"model": [ "gemma3", "deepseek-r1" ],
"tag": [ "ollama", "LoRA", "unsloth", "open-webui", "n8n" ]
}
```
## ollama
```sh
# mac
$ brew install ollama
$ brew services restart ollama
# windows
$ winget install ollama.ollama
$ ollama serve
$ ollama pull gemma3:1b
$ ollama run gemma3:1b "hello"
```
## n8n
```sh
# https://github.com/n8n-io/n8n/
$ docker volume create n8n_data
$ docker run -it --rm --name n8n -p 5678:5678 -v n8n_data:/home/node/.n8n docker.n8n.io/n8nio/n8n
```
## webui
```sh
$ winget install python.python.3.12
$ python --version
$ python -m venv webui
$ cd webui
$ .\Scripts\activate
$ pip install open-webui
$ open-webui serve
http://localhost:8080
```
## LoRA
apple siliconでLoRA(finetuning)するには`mlx_lm`を使用します。
```sh
$ brew install --cask anaconda
$ brew info anaconda
$ cd /opt/homebrew/Caskroom/anaconda/*
$ ./Anaconda3*.sh
```
`google/gemma-3-1b-it`を承認しておきます。
- https://huggingface.co/google/gemma-3-1b-it
```sh
$ pip install -U "huggingface_hub[cli]"
# https://huggingface.co/settings/tokens
# Repositories permissions:Read access to contents of selected repos
$ huggingface_hub login
```
```sh
$ conda create -n finetuning python=3.12
$ conda activate finetuning
$ pip install mlx-lm
$ echo "{ \"model\": \"https://huggingface.co/google/gemma-3-1b-it\", \"data\": \"https://github.com/ml-explore/mlx-examples/tree/main/lora/data\" }"|jq .
$ git clone https://github.com/ml-explore/mlx-examples
$ model=google/gemma-3-1b-it
$ data=mlx-examples/lora/data
$ mlx_lm.lora --train --model $model --data $data --batch-size 3
$ ls adapters
adapter_config.json
adapters.safetensors
```
## unsloth
windowsでLoRA(finetuning)するには`unsloth`を使います。
最も安定しているのは以下のverです。
```sh
$ nvidia-smi
$ nvcc --version
# https://github.com/unslothai/notebooks/blob/main/unsloth_windows.ps1
cuda: 12.4
python: 3.11
```
torchをcuda:12.8で使用する方法もあります。python:3.12を使用できます。しかし、unslothのinstallでは問題が発生しました。
```sh
$ winget install --scope machine nvidia.cuda --version 12.4.1
$ winget install curl.curl
```
```sh
# https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation
$ curl -sLO https://raw.githubusercontent.com/unslothai/notebooks/refs/heads/main/unsloth_windows.ps1
$ powershell.exe -ExecutionPolicy Bypass -File .\unsloth_windows.ps1
$ vim custom.py
```
上記はpwshでunsolthを使う方法ですが、wslを使ったほうがいいです。
```py
# https://docs.unsloth.ai/get-started/fine-tuning-guide
from unsloth import FastModel
import torch
fourbit_models = [
# 4bit dynamic quants for superior accuracy and low memory use
# https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-tune-gemma-3
# https://huggingface.co/unsloth/gemma-3-4b-it
"unsloth/gemma-3-1b-it-unsloth-bnb-4bit",
"unsloth/gemma-3-4b-it-unsloth-bnb-4bit",
"unsloth/gemma-3-12b-it-unsloth-bnb-4bit",
"unsloth/gemma-3-27b-it-unsloth-bnb-4bit",
# Other popular models!
"unsloth/Llama-3.1-8B",
"unsloth/Llama-3.2-3B",
"unsloth/Llama-3.3-70B",
"unsloth/mistral-7b-instruct-v0.3",
"unsloth/Phi-4",
] # More models at https://huggingface.co/unsloth
model, tokenizer = FastModel.from_pretrained(
model_name = "unsloth/gemma-3-4b-it",
max_seq_length = 2048, # Choose any for long context!
load_in_4bit = True, # 4 bit quantization to reduce memory
load_in_8bit = False, # [NEW!] A bit more accurate, uses 2x memory
full_finetuning = False, # [NEW!] We have full finetuning now!
# token = "hf_...", # use one if using gated models
)
model = FastModel.get_peft_model(
model,
finetune_vision_layers = False, # Turn off for just text!
finetune_language_layers = True, # Should leave on!
finetune_attention_modules = True, # Attention good for GRPO
finetune_mlp_modules = True, # SHould leave on always!
r = 8, # Larger = higher accuracy, but might overfit
lora_alpha = 8, # Recommended alpha == r at least
lora_dropout = 0,
bias = "none",
random_state = 3407,
)
```
## comfyui
https://github.com/comfyanonymous/comfyui
- https://github.com/ltdrdata/comfyui-manager
- https://github.com/ltdrdata/comfyui-impact-pack
開発者のmatrix roomが`comfyui_space:matrix.org`にあります。
https://app.element.io/#/room/#comfyui_space:matrix.org
### install
installは`git clone`から構築したほうがいいです。
```sh
# https://docs.comfy.org/installation/manual_install#nvidia:install-nightly
$ winget install python.python.3.12
$ git clone https://github.com/comfyanonymous/comfyui
$ cd comfyui
$ python -m venv venv
$ Set-ExecutionPolicy RemoteSigned -Scope Process
$ venv\Scripts\activate
$ pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128
$ python -m pip install --upgrade pip
$ pip install -r requirements.txt
$ python main.py
http://localhost:8188
```
comfyuiはnodeで書きますが`workflow`と呼ぶようです。jsonで保存されます。簡単にimportできます。
https://comfyanonymous.github.io/ComfyUI_examples/
基本構造としては、以下のとおりです。
```sh
./comfyui
├── main.py
├── custom_nodes # ここにpluginを入れる
│   └── comfyui-manager
└── models
└── checkpoints # ここにmodelを入れる
└── model.safetensors
```
1. modelは[comfyui-manager](https://github.com/ltdrdata/comfyui-manager)、または[civitai.com](https://civitai.com/models)からdlするといいです。
2. workflowは[example](https://github.com/aimpowerment/comfyui-workflows)があります。また、[openart.ai](https://openart.ai/workflows/all)と連携できます。
3. promptは[majinai.art](https://majinai.art/ja/)を参照してください。
例えば、`text-to-image.json`をworkflowに読み込んで、modelとpromptを書き換えて生成してみます。
```sh
# https://docs.comfy.org/get_started/first_generation
$ curl -sLO https://raw.githubusercontent.com/Comfy-Org/docs/refs/heads/main/public/text-to-image.json
```
内容は以下のとおりです。jsonはimport(workflow)用ではありません。
```json
{
"models": {
"checkpoints": "SD2.1/wd-illusion-fp16.safetensors",
"vae": "SD2.1/kl-f8-anime2.ckpt"
},
"prompt": {
"positive": "(little girl, head, gold hair, long, violet eyes :1.1), 1girl, solo, gold hair, long hair, galaxy background, looking at viewer, (depth of field, blurry, blurry background, bokeh:1.4), white dress, angel halo",
"negative": "nsfw, (worst quality, low quality:1.4)"
}
}
```
![](https://git.syui.ai/ai/ai/raw/branch/main/repos/comfyui/output/ComfyUI_00001_.png)
### comfyui + torch + cuda:12.8
`cuda:12.8`で問題が発生した場合、こちらを参考にしてください。
私の環境だと問題は発生しませんでした。
```sh
$ cd ComfyUI/.venv/Scripts/
$ ./python.exe -V
python:3.12
```
`torch`のnightly versionは`cuda:12.8`に対応しています。
https://pytorch.org/get-started/locally/
```sh
$ pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128
```
しかし、`torchaudio`などで衝突が発生します。衝突が発生する場合はversionを指定します。しかし、この方法で互換性が完全に解消されるわけではありません。
```sh
$ ./python.exe -m pip uninstall torch torchvision torchaudio -y
$ ./python.exe -m pip install --pre torch==2.7.0.dev20250306+cu128 torchvision==0.22.0.dev20250306+cu128 torchaudio==2.6.0.dev20250306+cu128 --index-url https://download.pytorch.org/whl/nightly/cu128
```
ホイールファイルを使用すると安定するようです。
```sh
# https://huggingface.co/w-e-w/torch-2.6.0-cu128.nv
$ ./python.exe -m pip install torch-2.x.x+cu128-cp312-cp312-win_amd64.whl
$ ./python.exe -m pip install torchvision-x.x.x+cu128-cp312-cp312-win_amd64.whl
$ ./python.exe -m pip install torchaudio-x.x.x+cu128-cp312-cp312-win_amd64.whl
$ ./python.exe -c "import torch; print(torch.cuda.is_available()); print(torch.__version__); print(torch.cuda.get_arch_list())"
```
### comfyui + cosmos
nvidiaのcosmosを使った動画生成です。
https://comfyanonymous.github.io/ComfyUI_examples/cosmos/
### comfyui + ollama
- https://github.com/stavsap/comfyui-ollama
- https://github.com/pythongosssss/ComfyUI-Custom-Scripts
`show text`のcustom nodeを使用するには`ComfyUI-Custom-Scripts`が必要です。
https://github.com/aimpowerment/comfyui-workflows/blob/main/ollama-txt2img-workflow.json

BIN
icon/ai.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 140 KiB

BIN
repos/comfyui/output/ComfyUI_00001_.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 454 KiB

729
repos/comfyui/user/default/workflows/image_to_video_cosmos_7B.json Normal file
View File

@ -0,0 +1,729 @@
{
"last_node_id": 84,
"last_link_id": 198,
"nodes": [
{
"id": 38,
"type": "CLIPLoader",
"pos": [
-332.08502197265625,
231.04571533203125
],
"size": [
315,
82
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"links": [
75,
99
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPLoader"
},
"widgets_values": [
"oldt5_xxl_fp8_e4m3fn_scaled.safetensors",
"cosmos",
"default"
]
},
{
"id": 8,
"type": "VAEDecode",
"pos": [
1270.6927490234375,
120.51702117919922
],
"size": [
210,
46
],
"flags": {},
"order": 13,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 87
},
{
"name": "vae",
"type": "VAE",
"link": 76
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
172,
181
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecode"
},
"widgets_values": []
},
{
"id": 61,
"type": "ModelSamplingContinuousEDM",
"pos": [
410,
-20
],
"size": [
327.5999755859375,
106
],
"flags": {},
"order": 9,
"mode": 4,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 157
}
],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
194
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ModelSamplingContinuousEDM"
},
"widgets_values": [
"edm",
80,
0.002
]
},
{
"id": 74,
"type": "LTXVConditioning",
"pos": [
540,
150
],
"size": [
210,
78
],
"flags": {},
"order": 11,
"mode": 4,
"inputs": [
{
"name": "positive",
"type": "CONDITIONING",
"link": 185
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 186
}
],
"outputs": [
{
"name": "positive",
"type": "CONDITIONING",
"links": [
187
],
"slot_index": 0
},
{
"name": "negative",
"type": "CONDITIONING",
"links": [
188
],
"slot_index": 1
}
],
"properties": {
"Node name for S&R": "LTXVConditioning"
},
"widgets_values": [
24
]
},
{
"id": 80,
"type": "Note",
"pos": [
475.15997314453125,
-163.2658233642578
],
"size": [
266.2419128417969,
99.78375244140625
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"These pink nodes are \"bypassed\" meaning they don't do anything. To unbypass them: right click -> bypass\n\n"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 82,
"type": "Note",
"pos": [
-1.2908354997634888,
565.2498168945312
],
"size": [
312.01824951171875,
126.14599609375
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"The positive and negative prompts should be long. Short prompts will still generate a coherent video however it might not follow the prompt very well."
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 7,
"type": "CLIPTextEncode",
"pos": [
0,
330
],
"size": [
425.27801513671875,
180.6060791015625
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 75
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
186
],
"slot_index": 0
}
],
"title": "CLIP Text Encode (Negative Prompt)",
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"The video captures a series of frames showing ugly scenes, static with no motion, motion blur, over-saturation, shaky footage, low resolution, grainy texture, pixelated images, poorly lit areas, underexposed and overexposed scenes, poor color balance, washed out colors, choppy sequences, jerky movements, low frame rate, artifacting, color banding, unnatural transitions, outdated special effects, fake elements, unconvincing visuals, poorly edited content, jump cuts, visual noise, and flickering. Overall, the video is of poor quality."
],
"color": "#322",
"bgcolor": "#533"
},
{
"id": 69,
"type": "SaveAnimatedWEBP",
"pos": [
1520,
120
],
"size": [
763.5289916992188,
578.3422241210938
],
"flags": {},
"order": 14,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 172
}
],
"outputs": [],
"properties": {},
"widgets_values": [
"ComfyUI",
24,
false,
80,
"default"
]
},
{
"id": 73,
"type": "SaveAnimatedPNG",
"pos": [
2299.842041015625,
121.30572509765625
],
"size": [
720.1341552734375,
829.0499877929688
],
"flags": {},
"order": 15,
"mode": 4,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 181
}
],
"outputs": [],
"properties": {
"Node name for S&R": "SaveAnimatedPNG"
},
"widgets_values": [
"ComfyUI",
24,
4
]
},
{
"id": 3,
"type": "KSampler",
"pos": [
843.46337890625,
122.69183349609375
],
"size": [
385.8114318847656,
262
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 194
},
{
"name": "positive",
"type": "CONDITIONING",
"link": 187
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 188
},
{
"name": "latent_image",
"type": "LATENT",
"link": 196
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
87
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "KSampler"
},
"widgets_values": [
959521281192720,
"randomize",
20,
6.5,
"res_multistep",
"karras",
1
]
},
{
"id": 39,
"type": "VAELoader",
"pos": [
950.17138671875,
453.1830749511719
],
"size": [
278.68310546875,
58
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
76,
195
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAELoader"
},
"widgets_values": [
"cosmos_cv8x8x8_1.0.safetensors"
]
},
{
"id": 81,
"type": "Note",
"pos": [
480,
730
],
"size": [
332.6131591796875,
168.23121643066406
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"This model loves a length of 121 and anything too far away from this will result in a bad video.\n\nThe width and height should be equal or bigger to 704\n\nYou can set a start_image, end_image or both at the same time."
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 37,
"type": "UNETLoader",
"pos": [
7.319890975952148,
-20.895429611206055
],
"size": [
380,
82
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
157
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UNETLoader"
},
"widgets_values": [
"Cosmos-1_0-Diffusion-7B-Video2World.safetensors",
"default"
]
},
{
"id": 84,
"type": "LoadImage",
"pos": [
-6.688927173614502,
743.3736572265625
],
"size": [
416.1836242675781,
366.83038330078125
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
198
],
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": null
}
],
"properties": {
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"ComfyUI_256804_.png",
"image"
]
},
{
"id": 42,
"type": "CLIPTextEncode",
"pos": [
0,
120
],
"size": [
422.84503173828125,
164.31304931640625
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 99
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
185
],
"slot_index": 0
}
],
"title": "CLIP Text Encode (Positive Prompt)",
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"cute anime girl with massive fennec fox ears and a big fluffy tail long blonde wavy hair blue eyes wearing a pink plaid sweater and a red scarf with a oversized black open coat with a golden circuit board pattern and a long blue maxi skirt and large black boots standing in the beautiful outdoors snow with amazing view mountains forest sky clouds beautiful sunset evening colorful horizon, she is smiling as the evening turns into night\n\n"
],
"color": "#232",
"bgcolor": "#353"
},
{
"id": 83,
"type": "CosmosImageToVideoLatent",
"pos": [
480,
490
],
"size": [
315,
170
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "vae",
"type": "VAE",
"link": 195
},
{
"name": "start_image",
"type": "IMAGE",
"link": 198,
"shape": 7
},
{
"name": "end_image",
"type": "IMAGE",
"link": null,
"shape": 7
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
196
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CosmosImageToVideoLatent"
},
"widgets_values": [
1024,
1024,
121,
1
]
}
],
"links": [
[
75,
38,
0,
7,
0,
"CLIP"
],
[
76,
39,
0,
8,
1,
"VAE"
],
[
87,
3,
0,
8,
0,
"LATENT"
],
[
99,
38,
0,
42,
0,
"CLIP"
],
[
157,
37,
0,
61,
0,
"MODEL"
],
[
172,
8,
0,
69,
0,
"IMAGE"
],
[
181,
8,
0,
73,
0,
"IMAGE"
],
[
185,
42,
0,
74,
0,
"CONDITIONING"
],
[
186,
7,
0,
74,
1,
"CONDITIONING"
],
[
187,
74,
0,
3,
1,
"CONDITIONING"
],
[
188,
74,
1,
3,
2,
"CONDITIONING"
],
[
194,
61,
0,
3,
0,
"MODEL"
],
[
195,
39,
0,
83,
0,
"VAE"
],
[
196,
83,
0,
3,
3,
"LATENT"
],
[
198,
84,
0,
83,
1,
"IMAGE"
]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}

1
repos/comfyui/user/default/workflows/text2image.json Normal file
View File

@ -0,0 +1 @@
{"id":"b6495d7d-ade3-4f36-b2d2-591a9b63e63f","revision":0,"last_node_id":25,"last_link_id":39,"nodes":[{"id":5,"type":"EmptyLatentImage","pos":[417,533],"size":[315,106],"flags":{},"order":0,"mode":0,"inputs":[],"outputs":[{"localized_name":"LATENT","name":"LATENT","type":"LATENT","slot_index":0,"links":[2]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.27","Node name for S&R":"EmptyLatentImage"},"widgets_values":[576,576,1]},{"id":8,"type":"VAEDecode","pos":[1176.900390625,105.46100616455078],"size":[210,46],"flags":{},"order":6,"mode":0,"inputs":[{"localized_name":"samples","name":"samples","type":"LATENT","link":7},{"localized_name":"vae","name":"vae","type":"VAE","link":25}],"outputs":[{"localized_name":"IMAGE","name":"IMAGE","type":"IMAGE","slot_index":0,"links":[39]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.27","Node name for S&R":"VAEDecode"},"widgets_values":[]},{"id":25,"type":"SaveImage","pos":[1411,109],"size":[249.5288543701172,270],"flags":{},"order":7,"mode":0,"inputs":[{"localized_name":"images","name":"images","type":"IMAGE","link":39}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.27"},"widgets_values":["ComfyUI",""]},{"id":3,"type":"KSampler","pos":[785,121],"size":[315,262],"flags":{},"order":5,"mode":0,"inputs":[{"localized_name":"model","name":"model","type":"MODEL","link":18},{"localized_name":"positive","name":"positive","type":"CONDITIONING","link":4},{"localized_name":"negative","name":"negative","type":"CONDITIONING","link":6},{"localized_name":"latent_image","name":"latent_image","type":"LATENT","link":2}],"outputs":[{"localized_name":"LATENT","name":"LATENT","type":"LATENT","slot_index":0,"links":[7]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.27","Node name for S&R":"KSampler"},"widgets_values":[734512375191575,"randomize",15,7,"dpmpp_2m","karras",1]},{"id":16,"type":"CheckpointLoaderSimple","pos":[-24,263],"size":[301.42279052734375,98],"flags":{},"order":1,"mode":0,"inputs":[],"outputs":[{"localized_name":"MODEL","name":"MODEL","type":"MODEL","slot_index":0,"links":[18]},{"localized_name":"CLIP","name":"CLIP","type":"CLIP","slot_index":1,"links":[19,20]},{"localized_name":"VAE","name":"VAE","type":"VAE","slot_index":2,"links":[]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.27","Node name for S&R":"CheckpointLoaderSimple"},"widgets_values":["SD2.1\\wd-illusion-fp16.safetensors"]},{"id":21,"type":"VAELoader","pos":[783,430],"size":[315,58],"flags":{},"order":2,"mode":0,"inputs":[],"outputs":[{"localized_name":"VAE","name":"VAE","type":"VAE","slot_index":0,"links":[25]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.27","Node name for S&R":"VAELoader"},"widgets_values":["SD1.5\\vae-ft-mse-840000-ema-pruned.safetensors"]},{"id":7,"type":"CLIPTextEncode","pos":[316.0503234863281,321.2099914550781],"size":[423.4383239746094,165.73065185546875],"flags":{},"order":4,"mode":0,"inputs":[{"localized_name":"clip","name":"clip","type":"CLIP","link":20}],"outputs":[{"localized_name":"CONDITIONING","name":"CONDITIONING","type":"CONDITIONING","slot_index":0,"links":[6]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.27","Node name for S&R":"CLIPTextEncode"},"widgets_values":["beautiful detailed"],"color":"#223","bgcolor":"#335"},{"id":6,"type":"CLIPTextEncode","pos":[310,117],"size":[422.84503173828125,164.31304931640625],"flags":{},"order":3,"mode":0,"inputs":[{"localized_name":"clip","name":"clip","type":"CLIP","link":19}],"outputs":[{"localized_name":"CONDITIONING","name":"CONDITIONING","type":"CONDITIONING","slot_index":0,"links":[4]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.27","Node name for S&R":"CLIPTextEncode"},"widgets_values":["little girl, beautiful detailed, beautiful detailed eyes,hyper detailed,hyper quality,eyes, and hair is same color,beautifuly color,face,{{{{{her hair is becoming Angel Halo,butterfly,}}}}},{{{{1girl}}}}kawaii,,{{{high details, high quality}}},{{{back light}}},{{hair and clothes is flower}},{{{upper body}}} ,high quality,hair with body ,webbed dress , light particles,black background}}}r,{{floating}}1girl,small breast, marbling with hair and clothes, looking at viewer,{{original}},{{arm down}}, {{paper cutting}}, black background,{{{{{highres}}}},} hair with flower,hair with flower ,hair, wavy hair ,diffusion lighting, abstract,Butterfly with bodｙ,big top sleeves, floating"],"color":"#322","bgcolor":"#533"}],"links":[[2,5,0,3,3,"LATENT"],[4,6,0,3,1,"CONDITIONING"],[6,7,0,3,2,"CONDITIONING"],[7,3,0,8,0,"LATENT"],[18,16,0,3,0,"MODEL"],[19,16,1,6,0,"CLIP"],[20,16,1,7,0,"CLIP"],[25,21,0,8,1,"VAE"],[39,8,0,25,0,"IMAGE"]],"groups":[{"id":1,"title":"Txt2Img","bounding":[-53,28,1200,631],"color":"#a1309b","font_size":24,"flags":{}},{"id":2,"title":"Save Image","bounding":[1166,29,509,155],"color":"#3f789e","font_size":24,"flags":{}}],"config":{},"extra":{"ds":{"scale":1.2100000000000006,"offset":[333.4562347429309,45.011930468982996]}},"version":0.4}

663
repos/comfyui/user/default/workflows/text_to_video_cosmos_7B.json Normal file
View File

@ -0,0 +1,663 @@
{
"last_node_id": 82,
"last_link_id": 194,
"nodes": [
{
"id": 38,
"type": "CLIPLoader",
"pos": [
-332.08502197265625,
231.04571533203125
],
"size": [
315,
82
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"links": [
75,
99
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPLoader"
},
"widgets_values": [
"oldt5_xxl_fp8_e4m3fn_scaled.safetensors",
"cosmos",
"default"
]
},
{
"id": 8,
"type": "VAEDecode",
"pos": [
1270.6927490234375,
120.51702117919922
],
"size": [
210,
46
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 87
},
{
"name": "vae",
"type": "VAE",
"link": 76
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
172,
181
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecode"
},
"widgets_values": []
},
{
"id": 61,
"type": "ModelSamplingContinuousEDM",
"pos": [
410,
-20
],
"size": [
327.5999755859375,
106
],
"flags": {},
"order": 9,
"mode": 4,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 157
}
],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
194
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ModelSamplingContinuousEDM"
},
"widgets_values": [
"edm",
80,
0.002
]
},
{
"id": 39,
"type": "VAELoader",
"pos": [
950.17138671875,
453.1830749511719
],
"size": [
278.68310546875,
58
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
76
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAELoader"
},
"widgets_values": [
"cosmos_cv8x8x8_1.0.safetensors"
]
},
{
"id": 78,
"type": "EmptyCosmosLatentVideo",
"pos": [
473.05047607421875,
380.00341796875
],
"size": [
315,
130
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
193
]
}
],
"properties": {
"Node name for S&R": "EmptyCosmosLatentVideo"
},
"widgets_values": [
1280,
704,
121,
1
]
},
{
"id": 74,
"type": "LTXVConditioning",
"pos": [
540,
150
],
"size": [
210,
78
],
"flags": {},
"order": 10,
"mode": 4,
"inputs": [
{
"name": "positive",
"type": "CONDITIONING",
"link": 185
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 186
}
],
"outputs": [
{
"name": "positive",
"type": "CONDITIONING",
"links": [
187
],
"slot_index": 0
},
{
"name": "negative",
"type": "CONDITIONING",
"links": [
188
],
"slot_index": 1
}
],
"properties": {
"Node name for S&R": "LTXVConditioning"
},
"widgets_values": [
24
]
},
{
"id": 80,
"type": "Note",
"pos": [
475.15997314453125,
-163.2658233642578
],
"size": [
266.2419128417969,
99.78375244140625
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"These pink nodes are \"bypassed\" meaning they don't do anything. To unbypass them: right click -> bypass\n\n"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 37,
"type": "UNETLoader",
"pos": [
70,
-20
],
"size": [
315,
82
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
157
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "UNETLoader"
},
"widgets_values": [
"Cosmos-1_0-Diffusion-7B-Text2World.safetensors",
"default"
]
},
{
"id": 81,
"type": "Note",
"pos": [
475.4506530761719,
570.9951782226562
],
"size": [
312.01824951171875,
126.14599609375
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"This model loves a length of 121 and anything too far away from this will result in a bad video.\n\nThe width and height should be equal or bigger to 704"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 82,
"type": "Note",
"pos": [
-1.2908354997634888,
565.2498168945312
],
"size": [
312.01824951171875,
126.14599609375
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"The positive and negative prompts should be long. Short prompts will still generate a coherent video however it might not follow the prompt very well."
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 42,
"type": "CLIPTextEncode",
"pos": [
0,
120
],
"size": [
422.84503173828125,
164.31304931640625
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 99
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
185
],
"slot_index": 0
}
],
"title": "CLIP Text Encode (Positive Prompt)",
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"A crystalline waterfall stands partially frozen, its edges draped with translucent ice that catches the sunlight in prisms of blue and silver. Below, a half-frozen pool spreads out, bordered by delicate ice formations. Through the fresh snow, a red fox moves gracefully, its russet coat vibrant against the white landscape, leaving perfect star-shaped prints behind as steam rises from its breath in the crisp winter air. The scene is wrapped in snow-muffled silence, broken only by the gentle murmur of water still flowing beneath the ice.\n\n"
],
"color": "#232",
"bgcolor": "#353"
},
{
"id": 7,
"type": "CLIPTextEncode",
"pos": [
0,
330
],
"size": [
425.27801513671875,
180.6060791015625
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 75
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
186
],
"slot_index": 0
}
],
"title": "CLIP Text Encode (Negative Prompt)",
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"The video captures a series of frames showing ugly scenes, static with no motion, motion blur, over-saturation, shaky footage, low resolution, grainy texture, pixelated images, poorly lit areas, underexposed and overexposed scenes, poor color balance, washed out colors, choppy sequences, jerky movements, low frame rate, artifacting, color banding, unnatural transitions, outdated special effects, fake elements, unconvincing visuals, poorly edited content, jump cuts, visual noise, and flickering. Overall, the video is of poor quality."
],
"color": "#322",
"bgcolor": "#533"
},
{
"id": 69,
"type": "SaveAnimatedWEBP",
"pos": [
1520,
120
],
"size": [
763.5289916992188,
578.3422241210938
],
"flags": {},
"order": 13,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 172
}
],
"outputs": [],
"properties": {},
"widgets_values": [
"ComfyUI",
24,
false,
80,
"default"
]
},
{
"id": 73,
"type": "SaveAnimatedPNG",
"pos": [
2299.842041015625,
121.30572509765625
],
"size": [
720.1341552734375,
829.0499877929688
],
"flags": {},
"order": 14,
"mode": 4,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 181
}
],
"outputs": [],
"properties": {
"Node name for S&R": "SaveAnimatedPNG"
},
"widgets_values": [
"ComfyUI",
24,
4
]
},
{
"id": 3,
"type": "KSampler",
"pos": [
843.46337890625,
122.69183349609375
],
"size": [
385.8114318847656,
262
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 194
},
{
"name": "positive",
"type": "CONDITIONING",
"link": 187
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 188
},
{
"name": "latent_image",
"type": "LATENT",
"link": 193
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
87
],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "KSampler"
},
"widgets_values": [
959521281192720,
"randomize",
20,
6.5,
"res_multistep",
"karras",
1
]
}
],
"links": [
[
75,
38,
0,
7,
0,
"CLIP"
],
[
76,
39,
0,
8,
1,
"VAE"
],
[
87,
3,
0,
8,
0,
"LATENT"
],
[
99,
38,
0,
42,
0,
"CLIP"
],
[
157,
37,
0,
61,
0,
"MODEL"
],
[
172,
8,
0,
69,
0,
"IMAGE"
],
[
181,
8,
0,
73,
0,
"IMAGE"
],
[
185,
42,
0,
74,
0,
"CONDITIONING"
],
[
186,
7,
0,
74,
1,
"CONDITIONING"
],
[
187,
74,
0,
3,
1,
"CONDITIONING"
],
[
188,
74,
1,
3,
2,
"CONDITIONING"
],
[
193,
78,
0,
3,
3,
"LATENT"
],
[
194,
61,
0,
3,
0,
"MODEL"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.3513057093105383,
"offset": [
347.8132028514172,
200.3286418889474
]
}
},
"version": 0.4
}