Momwe mungapangire ma avatar enieni okhala ndi Stable Diffusion ndi ComfyUI

¿Momwe mungapangire ma avatar enieni okhala ndi Stable Diffusion + ComfyUI? Kupanga avatar yowona komanso yosasinthika yokhala ndi Stable Diffusion ndi ComfyUI ndicholinga chotheka kutheka, koma pamafunika luso komanso kulingalira bwino. Chofunikira ndikusunga chizindikiritso (mawonekedwe a nkhope, tsitsi, mawonekedwe) pomwe mumasiyana mosiyanasiyana, kuyatsa ndi mawonekedwe., zomwe nthawi zambiri zimafuna kuphatikizika kwa kayendetsedwe ka ntchito, ma node enieni, ndipo nthawi zina zitsanzo zothandizira monga LoRA kapena zoyikapo.

Ogwiritsa ntchito ambiri amakumana ndi vuto lomwelo: ndi chithunzi chofotokozera, amapeza kufanana kwabwino pakuwombera kumodzi, koma chotsatira, mawonekedwe atsitsi kapena mtundu wamaso amasintha. Mwamvapo za embedding (textual inversion), LoRA, ndi ControlNet, ndipo ndizabwinobwino kudabwa kuti ndi njira iti yomwe ili yoyenera kwa inu.; Kuphatikiza apo, zosankha monga IP-Adapter ndi InstantID zikupitilizabe kuwonekera kuti zisinthe mawonekedwe a nkhope. M'nkhaniyi, tikuyankha mafunso omwe amafunsidwa kawirikawiri: kaya kutchulidwa kumodzi ndikokwanira, kaya kuli bwino kukonza LoRA kapena kugwiritsa ntchito zoyikapo, ndi ma node / makonzedwe omwe amagwira ntchito bwino mu ComfyUI kuti akwaniritse ma avatar okhazikika.

Kodi tikutanthauza chiyani ponena za kusasinthika mu avatar?

Tikamalankhula za kusasinthika, tikutanthauza kuti mawonekedwewo amakhalabe odziwika pazithunzi zingapo. Zimakhudza kusunga zofunikira (mawonekedwe a nkhope, maso, mphuno, milomo, tsitsi) ndi "kumverera" kwa phunzirolo. ngakhale timasewera ndi pose, kutsegula pakamwa, kuwala kolimba kapena maziko ovuta.

Kugwirizana uku kumachokera ku chidziwitso cha "anchoring" mumbadwo wamakono. Ngati chitsanzocho sichilandira zidziwitso zokwanira za yemwe mutuwo ndi ndani, chimakonda kusintha ndi kupatuka.; ndichifukwa chake ndizomveka kugwiritsa ntchito zowonera, ma module odziwika, kapena ma tweaks ang'onoang'ono (LoRA, embeddings) kuti mulimbikitse kufanana.

Kuphatikiza apo, ndikofunikira kulekanitsa zinthu zomwe zingasinthe popanda kuphwanya chizindikiritso komanso zomwe sizingatheke. Mbiri, zovala, mawonekedwe ndi njira zowunikira ndizosintha zotetezeka; mawonekedwe a maso, mtundu wa iris, tsitsi, ndi mapangidwe a mafupa, osati mochuluka. Kukonza malirewo ndi gawo lalikulu la ntchitoyo.

Kodi ndizotheka kukwaniritsa izi ndi chithunzi chimodzi mu ComfyUI?

Yankho lalifupi ndi: inde, ndi ma nuances. Chithunzi chimodzi chikhoza kukhala chokwanira ngati mutagwiritsa ntchito njira zowonetsera nkhope monga IP-Adapter (FaceID) kapena InstantID ndikuwongolera mlingo wa phokoso. mu img2img kapena mphamvu yokhazikika. Zowona, chithunzicho chiyenera kukhala chomveka bwino, chowala bwino, komanso chakutsogolo kapena chaching'ono, chokhala ndi mawonekedwe omveka bwino.

Ndi ComfyUI, njira yodziwika bwino ndikuphatikiza mawonekedwe a nkhope ndi liwiro lodziwika bwino komanso sampuli yokhazikika. Mawonekedwe owoneka "amakankhira" mtunduwo kuti ulemekeze mawonekedwe ake, pomwe chenjezo limalamula kalembedwe, maziko kapena kuyatsa.Ngati mukufuna kusiyanasiyana kosiyanasiyana, dalirani ControlNet (OpenPose) kuti muwongolere mawonekedwe osasokoneza nkhope.

Komabe, chithunzi chimodzi chili ndi malire ake: chikhoza "kuphunzira mopitirira" mawu enieni kapena kuwala kwa chithunzicho. Ngati mukuyang'ana kukhulupirika kopitilira muyeso komanso kusinthasintha, zithunzi zofotokozera za 6-20 zimasintha mawonekedwe., ndipo, ngati kuli kofunikira, LoRA yopepuka yophunzitsidwa pazithunzi zanu imapereka kusasinthika kwapamwamba kowombera.

Zoyika, LoRA, kapena Kukonza Bwino: Momwe Mungasankhire

Pali njira zitatu zazikuluzikulu zosinthira zidziwitso: zoyika (zosintha zamawu), LoRA, ndikusintha bwino. Kuyika kumaphunzitsa CLIP chizindikiro chatsopano chomwe chikuyimira mutu wanu., yokhala ndi MB zochepa komanso kuphunzitsidwa mwachangu, koma mphamvu zake ndizochepa poyerekeza ndi LoRA.

Kumbali ina, LoRA yophunzitsidwa bwino imalowetsa mphamvu mu zigawo za chitsanzo kuti igwire bwino kwambiri. Ndi 10-30 zithunzi zosiyanasiyana (makona, mawu, kuwala) ndi maphunziro apakati mungathe kukwaniritsa kusasinthasintha kwakukulu. mu SD 1.5 kapena SDXL, ndikusunga fayilo yaying'ono (makumi a MB). Awa ndiye malo okoma kwa ambiri.

Kukonzekera kwathunthu kwa poyang'anira kumasungidwa pazopanga zenizeni. Ndi okwera mtengo, deta zambiri, ndipo overwrites lonse kalembedwe chitsanzo.M'malo mwake, kwa ma avatar amunthu, LoRA yopepuka kapena mapaipi abwino olozera kumaso nthawi zambiri amakhala okwanira.

Ma node ndi midadada akulimbikitsidwa mu ComfyUI

Ma graph okhazikika amaphatikiza poyang'ana m'munsi, ma encoder a mawu, sampuli yokhazikika, ndi ma identity/control modules. Awa ndi midadada yothandiza kwambiri komanso momwe amasewerera limodzi:

• Checkpoint + VAE: Kwezani SD 1.5 kapena SDXL (kutengera zokongoletsa zanu ndi zomwe mumakonda). SDXL imapereka tsatanetsatane, koma imafuna VRAM yambiri.
• CLIP Text Encode (zabwino/zoipa): Zidziwitso zomveka, kutchula chizindikiro chamutu (ngati mukugwiritsa ntchito LoRA kapena kuyika) ndi kalembedwe / mawonekedwe.
• KSampler: DPM++ 2M Karras stable sampler, 20–35 masitepe, CFG 4–7 pa SDXL (6–9 pa SD1.5), mbewu yokhazikika kuti ibereke.
• IP-Adapter / InstantID: kukonza ndi nkhope kulimbikitsa makhalidwe; sinthani mphamvu (0.6-0.9) molingana ndi zopotoka.
• ControlNet (OpenPose/Depth/Canny): Kuwongolera mawonekedwe, voliyumu ndi mizere pomwe chidziwitso chimakhala chokhazikika ndi IP-Adapter/LoRA.
• Lora Loader: Lowetsani LoRA ya mutu wanu ndi zolemera za 0.6-1.0; ngati amasokoneza kalembedwe, kuchepetsa kulemera kapena downscale CFG.
• Img2Img / Tiling: Kwa kusiyana kofewa, gwiritsani ntchito denoise 0.2-0.45; zamtengo wapatali zimawononga kudziwika.

Pamaziko awa, kuphatikiza kokhazikika nthawi zambiri kumakhala: Mutu wa LoRA + FaceID IP-Adapter + Pose ControlNetLoRA imatanthauzira mawonekedwe, IP-Adapter imakonza zinthu zabwino, ndipo ControlNet imakupatsani ufulu wosintha mawonekedwe anu ndi kaimidwe.

Mayendedwe oyambira pang'onopang'ono (ComfyUI)

Poyambira, mutha kupanga othamanga ochepa, olimba. Zidzakuthandizani ngati mutayamba kuchokera ku malemba enieni kapena ngati mutasintha pang'ono pa chithunzi.:

1. Load Checkpoint (SDXL kapena SD1.5) ndi Sinthani VAE.
2. CLIP Text Encode (zabwino): Fotokozani nkhaniyo ndi chizindikiro chawo kapena, ngati palibe LoRA, ndi mbali: «wamkulu, tsitsi lalifupi lofiirira, maso obiriwira, nkhope yozungulira» + kalembedwe kake («chithunzi cha kanema, kuwala kofewa»).
3. CLIP Text Encode (zoipa): imaphatikizapo zinthu zakale zopewera ("kusawoneka bwino, kupunduka, zala zowonjezera, maso osagwirizana, mtundu wa tsitsi wolakwika").
4. Adapta ya IP / InstantID: Lumikizani chithunzi cholozera ndikukhazikitsa mphamvu zoyambira ku 0.75 (kusintha 0.6–0.9). Ngati mukugwiritsa ntchito chithunzi chimodzi chokha, chiduleni kumaso ndikuwonetsetsa kuti chikuwoneka bwino.
5. ControlNet Pose (chosasankha): tanthauzirani poyimira ngati mukufuna mawu/mawonekedwe osiyanasiyana osataya dzina.
6. KSampler: DPM++ 2M Karras, 28-32 masitepe, CFG 5.5-7 (SDXL: imakonda kutsika pang'ono CFG). Mbewu zokhazikika zofananira.
7. VAE Decode ndipo, ngati kuli kofunikira, a upscaler (4x-UltraSharp, ESRGAN, kapena SDXL Refiner kuti mudziwe zambiri).

Ngati muli ndi fayilo ya Chithunzi cha LORA, onjezani pamaso pa woyesa ndi kulemera kwa 0.8 (yambani kutsika ndikukwera mmwamba ngati kufanana kulibe). Ndi LoRA yolimba mutha kuchepetsa mphamvu ya IP-Adapter, kulola LoRA kuti agwire chizindikiritso ndi IP-Adapter "yolondola".

Parameters zomwe zimapanga kusiyana

Pamene ikukonzekera kusasinthasintha, kusintha kwa magawo ang'onoang'ono kumakhala kotsimikizika. Kuwongolera mphamvu yokhazikika, denoise ndi mbewu kumakupatsani kukhazikika kwenikweni:

• Dinani pa img2img: 0.2–0.45 imasunga mawonekedwe ndipo imalola kuyatsa kosiyanasiyana/kumbuyo. Kuchokera pa 0.55, chizindikiritso chimasungunuka.
• Mtengo wa CFGNgati chithunzicho "chikakamizidwa" ndikupotozedwa, tsitsani CFG; ngati chitsanzocho chikunyalanyaza kufulumira kwanu, kwezani ndi theka la mfundo.
• Sampler/Masitepe: DPM++ 2M Karras kapena SDE Karras yokhala ndi masitepe 24–32 nthawi zambiri imapereka zotsatira zofananira popanda zopangira.
• Mbewu: Imayika mbewu zofananira. Pakusinthasintha pang'ono, gwiritsani ntchito "mbewu yosinthika" yokhala ndi mphamvu ya 0.1-0.3.
• Kusintha: 768–1024 kumbali yayitali imathandizira mawonekedwe abwino a nkhope. Ku SDXL, 1024 ndiye malo okoma atsatanetsatane.

Ngati tsitsi kapena mtundu wa diso usintha, onjezani "mtundu wolakwika wa tsitsi, kusintha kwamtundu, mtundu wamaso wosagwirizana" molakwika ndikubwereza. Zimathandizanso kuwonetsa mtundu ngati gawo lachidziwitso chabwino pakuwombera kulikonse. kuteteza chitsanzocho kuti "chisayiwale".

Mawu, maziko ndi kuyatsa popanda kutaya chidziwitso

Pamawu osinthika (kumwetulira, kudabwa, tsegulani pakamwa), dalirani ControlNet OpenPose kapena, bwinobe, chowongolera cha zizindikiro zamaso zikapezeka. Kuwongolera ma geometry a nkhope kumachepetsa zopindika ndikulepheretsa mtunduwo kupanga mawonekedwe..

Powunikira, pangani ndondomekoyi momveka bwino: "bokosi lofewa kuchokera kumanzere", "kuwala kwamphepo", "ora lagolide". Kugwiritsa ntchito zolozera zachilengedwe (maganizidwe a HDRI, mafotokozedwe a studio) amawongolera mithunzi popanda kukhudza zomwe ziliNgati kamvekedwe ka khungu kakusintha, onjezani "kufanana kwa khungu" kapena ikani kutentha kwamtundu mwachangu.

Pazinthu zovuta, gwiritsani ntchito ControlNet Depth kapena Canny pa mphamvu yochepa (0.35-0.55) ndikufotokozera chilengedwe mwamsanga. IP-Adapter/LoRA iyenera kukhala yolemera kwambiri kuposa ControlNet yakumbuyo kotero kuti nkhope isaipitsidwe ndi mikombero yakunja.

Pamene mukufuna kusintha maonekedwe anu (zovala / zowonjezera), lowetsani malembawo ndikufewetsa kulemera kwa LoRA ngati nthawi zonse "kukoka" chovala chomwecho. Ma LoRA amatha kupitilira zokongoletsa; sungani masikelo kuti malangizo atsopano atumizidwe..

Kuphunzitsa kapena kusaphunzitsa: malangizo othandiza pa LoRA/zoyika

Ngati mawonekedwe a nkhope sakukwanira, ganizirani za LoRA ya nkhaniyi. Gwiritsani ntchito zithunzi 10-30 zokhala ndi makona osiyanasiyana, mawu, maziko, ndi zowunikira (koma nkhope yanu ikhale yaukhondo komanso yakuthwa).. Dulani mbali yaifupi kukhala 512–768 px, sungani mwamuna/amuna ngati maziko anu ndi a generalist, ndipo zindikirani dzina lachizindikiro.

Magawo ophunzitsira (SD1.5): udindo 4-8, alpha wofanana ndi udindo, mlingo wophunzirira 1e-4 mpaka 5e-5, masitepe 2k-6k ndi batch yaying'ono. Pewani kuphunzitsa mopambanitsa; ngati muwona "clone" ya chithunzi chimodzi, chepetsani masitepe kapena onjezerani zambiri.. Pa SDXL, gwiritsani ntchito malingaliro apamwamba ndikutenga VRAM yambiri.

Pazoyika (zolemba zolemba), zithunzi za 3-10 zitha kugwira ntchito, koma mudzafunika masitepe ambiri kuti mukhale bata. Zoyikapo sizimakhudza kukongola konsekonse ndipo zimalemera pang'ono., yabwino ngati mukufuna chizindikiro chogwiritsidwanso ntchito popanda kuyang'anira LoRA.

Quality, makulitsidwe ndi retouching

Chithunzi choyambira chikapangidwa, ikani 2-4x scaler (ESRGAN, 4x UltraSharp) kapena SDXL refiner kuti mudziwe zambiri za nkhope. Woyenga amatha kukonza khungu ndi maso popanda kuyambitsa zinthu zakale, makamaka ngati mumasunga mbewu komanso mwachangu.

Kuti mukonze maso / pakamwa, mutha kugwiritsa ntchito ADetailer kapena node zobwezeretsa nkhope. Konzani zolakwika zapafupi ndikusunga zolembedwa zinaPewani zosefera zankhanza zomwe "zimapanga pulasitiki" pakhungu; m'malo mwake, sinthani kuwongolera bwino komanso makonda a microcontrast.

Kuthetsa mavuto wamba

Ngati tsitsi la tsitsi likusintha pakati pa kutengera, vuto nthawi zambiri limakhala phokoso lambiri kapena zomveka. M'munsi denoise / CFG, limbitsani "tsitsi lalifupi labulauni" kapena tchulani tsitsi lapadera pa nthawi iliyonse. Ngati mugwiritsa ntchito LoRA, onjezerani kulemera kwake ndi 0.1.

Ngati maso amasiyana mtundu, onjezani "maso obiriwira, mtundu wamaso wokhazikika" ndikulemba "mtundu wamaso wosagwirizana, heterochromia" molakwika. IP-Adapter/InstantID imathandizanso ndi tsatanetsatane wa iris pamene umboniwo uli womveka bwino.

Ngati kalembedwe "kudya" chidziwitso (mwachitsanzo, kalembedwe kolimba LoRA), kuchepetsa kulemera kwake kapena kuonjezera kulemera kwa phunziro LoRA. Kulinganiza masikelo ndikofunikira kuti tipewe kufanana.Njira ina ndikutsitsa CFG kuti mtunduwo usakakamize kalembedwe kwambiri.

Ngati kusiyanasiyana kuli kochepa, onjezerani pang'ono phokoso (0.05-0.1) kapena gwiritsani ntchito mbewu zosintha. Kukankhira pang'ono kwachisawawa kumapanga zosiyanasiyana popanda kuphwanya mawonekedwe.

Madera ndi Miyezo: Komwe Mungaphunzire ndi Kugawana

Gulu la Stable Diffusion pa Reddit ndi lalikulu komanso lachangu kwambiri. Mu / r/StableDiffusion mutha kutumiza zojambulajambula, kufunsa mafunso, kukambirana, ndikuthandizira njira zatsopano zotseguka.; Sibwalo lovomerezeka, koma mzimu wake ndikuthandizira chilengedwe chotseguka ndikukuthandizani kuti muchite bwino.

ComfyUI subreddit, nawonso ammudzi/osavomerezeka, ndi malo abwino ogawana ntchito, mafunso, ndi malangizo. Chonde sungani zolemba za SFW, musalimbikitse mitsinje yolipidwa, khalani pamutu, ndipo koposa zonse, khalani okoma mtima.Kunyalanyaza zotsatira za anthu ena kudzaletsa, ndipo tikulimbikitsidwa kuti musasokoneze chakudya chanu ndi ma post ambiri motsatizana.

Kuwona ulusi komwe ma graph ndi magawo amalumikizidwa ndi njira yabwino yofulumizitsira kuphunzira kwanu. Kuwona ma benchmark okhala ndi mbewu zokhazikika, zolemetsa za LoRA, ndi zithunzi zolozera kumakuwonetsani makonda omwe amagwira ntchito. mukuchita.

Kuchokera pa chithunzi kupita ku kanema wokhala ndi mawu: StableAvatar

Ngati mukufuna kupita patsogolo ndikukhala ndi avatar yomwe "imalankhula" pogwiritsa ntchito mawu, onani StableAvatar. Ndi chimango chopangira mavidiyo amutu odalirika kwambiri, osasinthasintha kwakanthawi, omwe angakhale aatali opanda malire., kuyambira nyimbo zomvera.

Malinga ndi olemba ake, pagawo la 5-sekondi pa 480x832 ndi 25 fps, mtundu woyambira wokhala ndi -GPU_memory_mode=»model_full_load» umafunika pafupifupi 18 GB ya VRAM ndipo imatha pafupifupi mphindi 3 pa 4090 GPU. Izi zimapereka lingaliro lomveka bwino lazinthu zomwe zimafunikira komanso momwe zingathere pa hardware yamakono.. Khodi ndi mtundu zikupezeka pa: https://github.com/Francis-Rings/StableAvatar

Gululo likupita patsogolo kuti pakhale LoRA/finetuning mwachindunji padongosolo. Izi zimatsegula chitseko kuti mupitirize kusintha avatar ndi mawonekedwe ake a nkhope., zodziwikiratu monga momwe timachitira muzithunzi zosasunthika, koma m'makanema ogwirizana.

Mayankho achindunji a mafunso ofunika atatuwo

1) Kodi ndingapange ma avatar osasinthika mwachindunji ku ComfyUI ndi chithunzi chongoyerekeza? Inde, kugwiritsa ntchito IP-Adapter (FaceID) kapena InstantID ndi kuyenda kwamphamvu ndi phokoso lolamulirika ndi mbewu yokhazikika. Chithunzicho chiyenera kukhala chomveka komanso chakutsogolo; ndi kutchulidwa kumodzi pali malire ku kusiyana kwakukulu, koma kwa zithunzi ndi kusintha kwapakatikati zimagwira ntchito bwino kwambiri.

2) Kodi ndiyenera kulingalira kukonza bwino kapena kugwiritsa ntchito zoyika? Ngati mukuyang'ana kulimba kwambiri pazithunzi zambiri, mutu wopepuka wa LoRA ndiye njira yabwino kwambiri. khama / chiŵerengero chabwino cha zotsatiraZoyikapo (zosintha zamawu) ndizopepuka, koma zimagwira ma nuances ochepa. Kukonza kokwanira sikofunikira kaŵirikaŵiri kupatulapo zopanga zenizeni.

3) Kodi kasinthidwe ka node kapena njira zomwe ComfyUI zikuyenera kukhala ziti? Checkpoint + VAE + CLIP Text Encode (pos/neg) + KSampler (DPM++ 2M Karras, masitepe 24–32, CFG 5–7) + IP-Adapter/InstantID + ControlNet (ima/kuya kutengera malo). Kwezani LoRA ya mutuwo ndi kulemera kwa 0.6-1.0 ndi kuchepetsa mphamvu ya IP-Adapter pang'ono kuti onse azigwirizana.

4) Kodi Stable Diffusion imatanthauza chiyani ndipo ndi chiyani? Tikukuuzani zambiri m'nkhaniyi.

Musaiwale kuti magulu a /r/StableDiffusion ndi ComfyUI ndi malo otseguka momwe mungagawire zitsanzo, kufunsa mayankho, ndikupeza zanzeru zatsopano. Sungani zomwe muli nazo SFW, pewani kutsatsa zolipira, ndipo samalani ndi kamvekedwe kanu ndi omwe angoyamba kumene.; pakati pa onsewo, mlingo umakwera mofulumira kwambiri.

Ndi poyambira bwino (IP Adapter/Instant ID), mbewu yokhazikika, malangizo omveka bwino, ndi kuwongolera kamvekedwe kake, tsopano mutha kupeza zithunzi zofananira posintha masinthidwe, manja, ndi kuyatsa. Ngati muphunzitsanso LoRA yokhala ndi zithunzi 10-30 zosiyanasiyana, kufanana kumakula kwambiri., ndipo ndikuchita, kukonza bwino ControlNet ndi post-processing kukupatsani zotsatira zolimba ngakhale pazosankha zapamwamba. Kwa iwo omwe akufuna kupititsa patsogolo zinthu, StableAvatar ikuwonetsa kuti lingaliro lomwelo lachidziwitso chokhazikika lingagwiritsidwe ntchito pavidiyo yoyendetsedwa ndi ma audio ndi zida zoyenera.