more recent awemer version

does anyone have that script for downloading livestreams?

>>3759
here
but it got stuck on me once
i have to look into it

@lives recorder anon
it would be cool if the program output the .flv link into the console so I could copy paste them to the thread :)
Also, it is working very well, been running for three days straight with no issues

>>3950
you can already get it in two ways
* with -sm and the .json file will have it but the best quality url is hidden in json encoded "stream_data" tag
* ffmpeg output in .out file also has it at the start of the file

But i'll add it the next time i work on it no problem

>>4005
It's merely a convenience thing but I appreciate it anonie

>>4005
first issue I have received
File "E:\Downloads\awemer\tt_lives_recorder-main\tt_lives_recorder.py", line 144, in main
with urllib.request.urlopen(request) as url:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 216, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 519, in open
response = self._open(req, data)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 1352, in do_open
r = h.getresponse()
^^^^^^^^^^^^^^^
File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

>>4419
Looks like TikTok api error. If it keeps executing like normal then there's really nothing to do about it.
It should say somewhere "the script will keep executing"

>>4152
changelog:
* started versioning. assuming first upload was 1.0.0, the one after that 1.1.0 then this is 1.1.1
* fixed(hopefully) a bug where it would get perma stuck on api call (it might be a linux only thing)
* added an example of -f in -h
* added a python version check and meaningful output if the version is too old
* added a "best effort" cookie validity check. has potential for many false positives but all it does is log a warning
* -rt is now 5 by default (was 0). doesn't make sense to disable it, but you can if you want to for whatever reason
* added .flv url in log line when a new live is detected
* added -t/--trace for very verbose logging
* added -q/--quality flag if anybody wants lower quality streams for whatever reason


Also i need a better name for this. I was thinking of "livemer" or "livempy" but this has little in common with awemer.

I will work on tests now before I add any major changes since regression is a thing.

>>5529
cool, thanks for the updates
you could call it flver (pronounced flavor)

anyone else getting severely rate limited?

>>5529
I'm getting an "Api response is logically empty. Cookie might be expired." message quite a bit, even after getting a fresh cookie. Is my account getting cucked or could these be false positives?

>>6835
If your account has nothing going on in your inbox (messages, follow requests, activities) then it's a false positive. Other than calling some other api there is better way to check for cookie unfortunately.

>>6729
a lot more details needed for a reasonable answer

It asks me for a sec_uid when trying to download a user's videos, where do I get that from?

>>8443
sorry I got it

@awemer anon, if you take suggestions
-add flag for downloading user avatar
-add flag for saving video metadata to file, just dump the api response data to a .info.json file, better compared to the description in file name

-add flag for applying video date as mtime to downloaded video file

getting empty response for t=uv
any relaters?

>>10596
yeah seems like it's broken

>>10626
its over

>>10726
any idea about t=uv?

>>10738
uv is working fine for me. try to clean/remame request_aweme_cache folder

>>10759
didn't help
what version/command are you using?

>>10768
this version >>2600
modified it here >>10726

node videos-node.js t=uv d=255 q=2 cookie=xxx l=10 i=6921421848353817605 f=./AwemerDownload

>>10775
got it working thanks
so they just started randomly validating device type?

>>10726
if anyone else is retarded enough to use the e command did it also break for you

>>10726
it's not working daamn
2023 sucks 10 dicks

>>11095
nevermind I did this and uv started working again
>>10759
thanks

awemer anon the e flag is no longer working for me with the new version

was anything changed about it?

How do I download both the h264 and h265 version?

>>11122
works for me

>>11604
could you post your commands?
I can get the liked json output file but when I run the command to scrape with it it just outputs another json

I haven't changed my commands at all, just using the newest awemer version

>>11122
input, with liked.json already having been made from a previous command: node videos-node.js o f e="t=uv n=0 j=sid_tt=xxx i={} g=@{}.txt l f=E:\downloads\liked q=2 z d=200 ui l convert_hevc" l d p=10 i=liked.json u;
output, which instead of being awemer downloading toks does this: output file: E:\Downloads\awemer\videos-node.js_20230107T000307Z_cmds.json

I am using the exact same batch file, just with the new version that fixed the empty response errors.

PLS

Some may be interested, yt-dlp added support for downloading Tiktok lives in its latest update
https://github.com/yt-dlp/yt-dlp/releases/tag/2023.01.06

I think I'm doing something wrong,
when I do:

J:\Archives\TikTok\awemer_2022.12.21>node videos-node.js t=v i=7179458700371840298 d o q=2 l metadata_each verify_files f=C:\folder

I get:
Error: Empty response.
at IncomingMessage.<anonymous> (J:\Archives\TikTok\awemer_2022.12.21\videos-node.js:896:14)
at IncomingMessage.emit (node:events:525:35)
at endReadableNT (node:internal/streams/readable:1358:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21)

This is using awemer_2022.12.21 and with the edited videos-node.js that was provided earlier. help?

>>11965
doesn't t=v require cookie to work?

>>11967
I've never used it with cookies and it's always worked for me. /gif/anon here btw (i really need a new nickname)

>>12194
tiktok has changed something. edited version works only with cookie=xxx

i know i said tests would be next but pythons mocking is kicking my ass and i've had these changes running for a while without issues so might aswell:

changelog:
* did major refactoring
script is now seperated into following files:
* arguments.py - argparse and config file parse logic
* constants.py - global and not so global constants
* coordination.py - stream recording state and it's control logic.
* environment.py - environment validation and setup logic
* post_process.py - post processing logic. converting merging and temporary file managing
* recording.py - flv recording logic
* utils.py - generic utils that don't fit anywhere else
* tt_lives_recorder.py stream detection/api invoking main loop logic
this should make the code easier to navigate if you want to look at it (which you should since i'm a stranger posting unverified code on the internet)
* added support for dual streams, will now record streams that someone you follow has joined
the end result might be you recording some random dude doing nothing for 5hours with your follower having joined for 5 seconds at the start
no flags to disable it for now
* added a filename mask for quality
* added a seperate log line for tiktok api errors and also a min wait so you can't spam it too much

also -mf with .mp4 files has been working pretty well for me even for very laggy streams

if anybody is still using this let me know how it's working for you and what options you're using

>11899
looking at the PR I have a suspicion it will not handle stream drops and will not work for all streams in the first place
if that's not the case let me know

>>13329
will try it later, thanks for your work recorderanon

>>13329
>which you should since i'm a stranger posting unverified code on the internet
I will not be doing this

where is the cookie id again i forgor

awemer is rigged with the bonbi virus, it will secretly download bonbi toks in your pc

anyone got a quick rundown of how awemer works?

>>14578
Here's your awemer guide

Download awemer and node.js
Install node and make sure "add to path" is ticked
Run install_dependencies.bat from awemer zip file

Get tokfu's user ID using a UID script like https://pastebin.com/HAqgGMvj (refresh their profile if it doesn't work at first, or look in F12 console)

Open the folder where videos-node.js is stored, click the directory bar, type "cmd" and hit enter
In the newly opened command window type something like this.
You need to fill in your own cookies, from Tiktok.com when you're logged in. Get the cookie by clicking the little lock next to the URL, or using EditThisCookie

node videos-node.js t=uv i=6584010823366344709 d o q=2 l=30 verify_files j=sid_tt=YOURCOOKIEFROMTIKTOK.COM;sessionid=THESAMECOOKIE f=D:\cutest_tokfu

>>14602
Oh and if you have trouble with install_dependencies you can skip it but you won't be able to use verify_files I think.

>>14602
no, actually i'm asking how the code works so that in the future i can try to fix it myself

>>14605
its a huge blob of js callbacks
very difficult to decipher
but you can try if you want the code is right there

>>14607
is it realistic if i try to rewrite it in python?

>>14625
realistic?
no
The scope is too big and the languages are too different.
You could do some basic version in realistic time-frames but the whole functionality of awemer is too vast imo

>>12276
sorry for the late reply. I usually use t=v but I tried using t=uv because t=v didn't work and gave me an "Empty response." error.

Is there any way of downloading stuff now without using cookies from a logged in account? I've never used cookies from tiktok as I don't have an account and it's worked perfectly fine for months.

>>16944
just create an account

>>16951
did that and its working fine

>>16959
only t=v is working though but i only cared about that

is there a way to get username/userid from a video id if the account is private?

>ignoring: Error: Empty response. | 1044
I'm so fucking tired of this shit

looks like t=v is broken again for me (Empty response.)

>>18098
nothing will happen unless t=uv breaks
thats when archive autists get involved

>>18098
just tested, it works for me

>>18885
that's off I'm still getting the same response even now. I'm using the latest awemer, what version are you using?

>>18886
odd *

>>18885
are you using a cookie?

i cant download tiktoks from this profile 7156516182881832235
works fine for all the other profiles ive tried

heres my script

node videos-node.js t=v d=255 q i=clipboard f="Videos\awemer" cookie=xxx convert_hevc rename_hevc progress_bars x

does it work for you guys?

>>18886
this edited version >>10726
the command is
node videos-node.js cookie=xxx t=v q=2 i=7181239952611544325 f=./download
>>18894
yes, it needs cookie for now

>>18895
fixed it
it didnt work because the source video is not in HD
switching to q=2 worked

how do i get userID?

>>18944
dl tampermonkey/violentmonkey extension
use this userscript

// UserScript
// @name Print UID to Console
// @namespace http://tampermonkey.net/
// @version 0.1
// @description try to take over the world!
// @author You
// @match https://www.tiktok.com/@* [Embed]
// @icon data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==
// @grant none
// /UserScript

(function() {
'use strict';

console.log('HELLO!');
//console.log(NEXT_DATA.props.pageProps.userInfo.user.id);
let userId = window['SIGI_STATE'].UserModule.users[window['SIGI_STATE'].UserPage.uniqueId].id;
console.log('HELLO!: ' + userId);
let infoNode = document.createElement("p");
infoNode.innerHTML = "Meta: " + userId;// + " " + NEXT_DATA.props.pageProps.userInfo.stats.videoCount;
//let descNode = document.getElementsByClassName("share-desc")[0];
//let descNode = document.getElementsByClassName("tiktok-b1wpe9-H2ShareDesc")[0];
let likesNode = document.getElementsByTagName("strong")[3];
likesNode.innerHTML = "Meta: " + userId;
//let containerNode = likesNode.parentNode;
//console.log(containerNode, containerNode.childElementCount);
//containerNode.insertBefore(infoNode, likesNode);
//console.log(containerNode, containerNode.childElementCount);
//console.log('DONE');
})();

>>18946
>>18947
i already tried this script and it didnt work
prints some red stuff in the console

>>18950
you have to look for the line that says "HELLO" and the UID
sometimes you have to refresh the page

>>18951
here are the outputs from print uid to console

HELLO! Print UID to Console.user.js:15:13
TypeError: window.SIGI_STATE.UserModule is undefined
VMwrdt7cwtq2p moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/ Print UID to Console.user.js#9:17
VMwrdt7cwtq2p moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/ Print UID to Console.user.js#9:30
VMwrdt7cwtq2p moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/ Print UID to Console.user.js#9:31
Ut moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
<anonymous> moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/ Print UID to Console.user.js#9:1
Se moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
post moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
ScriptData moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
onHandle moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
je moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
Print UID to Console.user.js:31:28
XHRGEThttps://www.tiktok.com/api/user/detail/?aid=1988&app_language=en&app_name=tiktok_web&browser_language=en-GB&browser_name=Mozilla&browser_online=true&browser_platform=Win32&browser_version=5.0 (Windows)&channel=tiktok_web&cookie_enabled=true&device_id=7188710928320693765&device_platform=web_pc&focus_state=true&from_page=user&history_len=3&is_fullscreen=false&is_page_visible=true&language=en&os=windows&priority_region=&referer=&region=EN&screen_height=1080&screen_width=1920&secUid=MS4wLjABAAAAs2aCZSPn0WmEbk351UV-pmM2TwrIh_2jCNaAR2KiiRCZuQsoC9wQsyzXG-b9NwWF&tz_name=Europe/Paris&uniqueId=jessepinkmanstinkman&webcast_language=en&msToken=JILdwxr4lJXAFB44o0SXx68eMOoJb8pP5VJjeNzLNVqLxlFrOmIpZfkJNdut6NFeaUYr2uGKCFdm-j7Ndfk4p1EVY8mgzcv_Zjqr6r75CyTIiPnBrWIys9pW36h_jvyW5XC56KhDOloDdw==&X-Bogus=DFSzsIVOy-iAN9Y/SZArz2O8kJMp&_signature=_02B4Z6wo000012MB99AAAIDCf0JJby6NiedjAPNAALsO08
[HTTP/2 200 OK 419ms]

>>18955
uhhhh make a new userscript and copy paste what I posted maybe? I have no idea what could be going wrong there.

>>18958
doesnt work
isnt there some other way of getting the userID? i remember you could type some cryptic shit into the console and get it that way, but i dont have it written anywhere.

i tried the extension on Edge and it worked thanks anyways

>>18961
you can also use this extension
https://mega.nz/folder/mIQizApQ#6zAAmcbzIy9dCIkvoOIN2Q

I didn't realize awempy broke cos I only ever scrape likes, but I've updated it to add the cookie flags
https://pastebin.com/fp17iaMf

>>13329
is this currently working? It seems like every cookie I generate gives "WARNING Api response is logically empty. Cookie might be expired.".

>>19975
just follow some account that is live and see if it records something

>>19980
I did, I get that error message.

>>19986
Maybe its your cookie/account then.
Works for me.
Can you scrape stories with that cookie?
That's not an error just a warning when the response is suspiciously empty. It can false positive if you're account doesn't have a lot going on.

Getting only empty responses even with cookie

>>20404
tikwm broken
sd should still work

>>20416
It's over

>>20428
should be back

Getting an internal server error.
Any idea what might be causing this? Not doing anything different than the last 20 times it worked.


Error: Internal Server Error
at ClientRequest.<anonymous> (C:\...\awemer_2022.12.10\videos-node.js:834:14)
at Object.onceWrapper (node:events:628:26)
at ClientRequest.emit (node:events:513:28)
at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:693:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:117:17)
at TLSSocket.socketOnData (node:_http_client:534:22)
at TLSSocket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:324:12)
at readableAddChunk (node:internal/streams/readable:297:9)
at Readable.push (node:internal/streams/readable:234:10)

>>22480
Seems a lot of people are having cookie issues, so I'll just add that I am using a logged in sessionid cookie.

Exact command;
node videos-node.js t=uv d=255 q=2 cookie=2361f490bbc89f8227722d10cd6481c6 l=5000 i=52025260371369984 f=./output

>>18946
>>18947
>>18950
>>18955
>>18961
fixed version
https://pastebin.com/raw/v9UHuwjb

Is there any neat greasemonkey scripts extensions that will just allow me to download a videos on an individual basis as I'm scrolling through a profiles videos. I tried a few on greasyfork but the best and most consistent I got was https://greasyfork.org/en/scripts/447369-tiktok-download . It works but you have to right click open in new tab which is a pain in the put.

>>22618
There's this one https://chrome.google.com/webstore/detail/tikfast-tiktok-video-down/enbpgklkminalibdialjfpkfejckhpfd

But it also opens into a new tab to select download options. It has a bulk feature but I've found it finicky.

>>22618
>>22619
don't you guys want to use this extension? >>18986
or do you not trust?

>>22625
If I'm being totally honest, I use this
https://ttdownloader.com/
For single video downloads, or even just a couple videos like when Lily updates.
It's not really what the other guy was asking for, so I didn't recommend it, but it works for me, I find it efficient, and don't mind the extra clicks

>>18986
nice extension
who made this, could you make github page for this extension?

>>18986

Nice, this is amazing

>>22682

https://github.com/giantcarrot/extension-description

Here ya go my nigger

>>22824
>>22682

Actually my bad the code isn't there. It was when I bookmarked it, sorry.

For the first time in weeks I'm getting the error again, despite not scrapping anything for 90 mins. Feels bad, man.