Comments (13)
Import completed successfully with ~30k messages.
from slack-to-discord.
Nice job, btw -- this project's code is far cleaner (and thus not off-putting to tweak) than other projects for doing similar things.
from slack-to-discord.
I made a quick hack to get around this, and then got another error:
Traceback (most recent call last):
File "/usr/local/bin/slack-to-discord", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.11/site-packages/slack_to_discord/__main__.py", line 27, in main
run_import(
File "/usr/local/lib/python3.11/site-packages/slack_to_discord/importer.py", line 569, in run_import
raise client._exception
File "/usr/local/lib/python3.11/site-packages/slack_to_discord/importer.py", line 412, in on_ready
await self._run_import(g)
File "/usr/local/lib/python3.11/site-packages/slack_to_discord/importer.py", line 480, in _run_import
for msg in slack_channel_messages(self._data_dir, chan_name, self._users, emoji_map, pins):
File "/usr/local/lib/python3.11/site-packages/slack_to_discord/importer.py", line 176, in slack_channel_messages
text = d["text"]
~^^^^^^^^
KeyError: 'text'
After a second work-around, the import is back up and running. Here's the diff for the hacks:
diff --git a/slack_to_discord/importer.py b/slack_to_discord/importer.py
index b83dc74..80acea1 100644
--- a/slack_to_discord/importer.py
+++ b/slack_to_discord/importer.py
@@ -142,6 +142,10 @@ def slack_filedata(f):
}
+def ts_fun(x):
+ if "ts" in x:
+ return x["ts"]
+
def slack_channel_messages(d, channel_name, users, emoji_map, pins):
def mention_repl(m):
type_ = m.group(1)
@@ -168,7 +172,9 @@ def slack_channel_messages(d, channel_name, users, emoji_map, pins):
for file in sorted(glob.glob(os.path.join(channel_dir, "*.json"))):
with open(file, "rb") as fp:
data = json.load(fp)
- for d in sorted(data, key=lambda x: x["ts"]):
+ for d in sorted(data, key=ts_fun):
+ if not "text" in d:
+ continue
text = d["text"]
text = MENTION_RE.sub(mention_repl, text)
text = LINK_RE.sub(lambda x: x.group(1), text)
from slack-to-discord.
So far so good ... the importer is powering though YEARS of Slack messages (still). Haven't had to restart since the above hack was put in place.
from slack-to-discord.
Glad everything worked for you!
Would it be possible for you to post or privately email me some (optionally redacted) JSON messages that don't have the ts
/text
fields in them? Since the ts
is a timestamp I'm really curious about what sort of message wouldn't have it. For text
, I've definitely seen events not have any, but usually it's just been an empty string/null
, not just be missing the field entirely.
Essentially I'd like to be able to understand why these cases exist in order to figure out the best way of dealing with them.
from slack-to-discord.
Yeah, that's the right way to do it.
I'll invert the logic, add a short-circuit, and then re-run: it should spit out the culprits pretty quickly.
I'll paste/attach sanitised data here ...
from slack-to-discord.
Were you ever able to re-run with more logging enabled? In case it was a matter of not wanting to deal with duplicate messages being imported or figuring out how to stub the code out, here's a simple patch that will just log out the offending messages without importing anything (applies to v1.1.5 - the latest release).
It can be automatically applied by saving it to a file, then running patch <path to slack_to_discord/importer.py> <path to patch>
in the terminal.
diff --git a/slack_to_discord/importer.py b/slack_to_discord/importer.py
index b83dc74..c6a73c9 100644
--- a/slack_to_discord/importer.py
+++ b/slack_to_discord/importer.py
@@ -168,6 +168,9 @@ def mention_repl(m):
for file in sorted(glob.glob(os.path.join(channel_dir, "*.json"))):
with open(file, "rb") as fp:
data = json.load(fp)
+ for x in data:
+ if "ts" not in x or "text" not in x:
+ print(x) # print out problematic message
for d in sorted(data, key=lambda x: x["ts"]):
text = d["text"]
text = MENTION_RE.sub(mention_repl, text)
@@ -260,6 +263,7 @@ def mention_repl(m):
else:
messages[ts] = msg
+ return # prevent actually importing anything
# Sort the dicts by timestamp and yield the messages
for msg in (messages[x] for x in sorted(messages.keys())):
msg["replies"] = [msg["replies"][x] for x in sorted(msg["replies"].keys())]
from slack-to-discord.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/slack_to_discord/importer.py", line 408, in on_ready
await self._run_import(g)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/slack_to_discord/importer.py", line 476, in _run_import
for msg in slack_channel_messages(self._data_dir, chan_name, self._users, emoji_map, pins):
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/slack_to_discord/importer.py", line 171, in slack_channel_messages
for d in sorted(data, key=lambda x: x["ts"]):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/slack_to_discord/importer.py", line 171, in <lambda>
for d in sorted(data, key=lambda x: x["ts"]):
~^^^^^^
KeyError: 'ts'
any solution to this @oubiwann @pR0Ps @maur1th
from slack-to-discord.
any solution to this
@syedzainqadri if you could apply the patch I posted above and report what it prints it would allow me to figure out why the issue is happening and how to fix it.
Alternatively, if you send me the export that's having issues (feel free to email it if it's sensitive, check my profile) I can figure it out from there.
from slack-to-discord.
Hi, I'm encountering the same error. Would be happy to email you the files in question.
here's the output
2024-01-17 23:05:26 INFO slack_to_discord.importer Processing channel '#techtalk'...
2024-01-17 23:05:26 CRITICAL slack_to_discord.importer Failed to finish import!
Traceback (most recent call last):
File "/Users/damon/.pyenv/versions/3.9.1/lib/python3.9/site-packages/slack_to_discord/importer.py", line 408, in on_ready
await self._run_import(g)
File "/Users/damon/.pyenv/versions/3.9.1/lib/python3.9/site-packages/slack_to_discord/importer.py", line 476, in _run_import
for msg in slack_channel_messages(self._data_dir, chan_name, self._users, emoji_map, pins):
File "/Users/damon/.pyenv/versions/3.9.1/lib/python3.9/site-packages/slack_to_discord/importer.py", line 171, in slack_channel_messages
for d in sorted(data, key=lambda x: x["ts"]):
File "/Users/damon/.pyenv/versions/3.9.1/lib/python3.9/site-packages/slack_to_discord/importer.py", line 171, in <lambda>
for d in sorted(data, key=lambda x: x["ts"]):
KeyError: 'ts'
2024-01-17 23:05:26 INFO slack_to_discord.importer Bot logging out
from slack-to-discord.
I'm using v 1.1.5 via pip. I applied the patch from 8/4/23, but still throws the error
from slack-to-discord.
I made a quick hack to get around this, and then got another error:
Traceback (most recent call last): File "/usr/local/bin/slack-to-discord", line 8, in <module> sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.11/site-packages/slack_to_discord/__main__.py", line 27, in main run_import( File "/usr/local/lib/python3.11/site-packages/slack_to_discord/importer.py", line 569, in run_import raise client._exception File "/usr/local/lib/python3.11/site-packages/slack_to_discord/importer.py", line 412, in on_ready await self._run_import(g) File "/usr/local/lib/python3.11/site-packages/slack_to_discord/importer.py", line 480, in _run_import for msg in slack_channel_messages(self._data_dir, chan_name, self._users, emoji_map, pins): File "/usr/local/lib/python3.11/site-packages/slack_to_discord/importer.py", line 176, in slack_channel_messages text = d["text"] ~^^^^^^^^ KeyError: 'text'After a second work-around, the import is back up and running. Here's the diff for the hacks:
diff --git a/slack_to_discord/importer.py b/slack_to_discord/importer.py index b83dc74..80acea1 100644 --- a/slack_to_discord/importer.py +++ b/slack_to_discord/importer.py @@ -142,6 +142,10 @@ def slack_filedata(f): } +def ts_fun(x): + if "ts" in x: + return x["ts"] + def slack_channel_messages(d, channel_name, users, emoji_map, pins): def mention_repl(m): type_ = m.group(1) @@ -168,7 +172,9 @@ def slack_channel_messages(d, channel_name, users, emoji_map, pins): for file in sorted(glob.glob(os.path.join(channel_dir, "*.json"))): with open(file, "rb") as fp: data = json.load(fp) - for d in sorted(data, key=lambda x: x["ts"]): + for d in sorted(data, key=ts_fun): + if not "text" in d: + continue text = d["text"] text = MENTION_RE.sub(mention_repl, text) text = LINK_RE.sub(lambda x: x.group(1), text)
I ended up using this workaround and it fixed the problem.
from slack-to-discord.
@damonseeley Thanks for sending me some sample data. The latest version (v1.1.6) has the fix in it.
from slack-to-discord.
Related Issues (20)
- Problem with the Guild HOT 3
- Is it possible to change the author of the message? HOT 4
- Issue importing HOT 1
- Add CLI option to control date separators
- Shard ID None heartbeat blocked HOT 5
- Maximum number of active threads reached HOT 5
- Thank you for making this!
- Clearly identify what `guild` is
- Failed to finish import - KeyError 'mode' HOT 3
- a HOT 1
- a
- Message formatting is lost when splitting large messages HOT 1
- Cannot install on macos 11.6 with Python 3.6.8 HOT 3
- After installation, script fails on import attempt HOT 1
- FileNotFoundError users.json HOT 4
- issue: Exception: Guild 'nameServer' not accessible to the bot. Available guild(s): HOT 2
- Error while importing one specific (public) channel HOT 8
- Improve migration process for large files HOT 1
- [Suggestions] Much faster imports
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from slack-to-discord.