There are still errors happening frequently, recently it's almost everyday. This is the log for one error but there may be different ones. Airflow logs get purged too quickly so they're hard to catch.
[2020-10-24 04:21:07,743] {logging_mixin.py:112} INFO - [2020-10-24 04:21:07,743] {pod_launcher.py:125} INFO - b'INFO:root:Creating external table tmp.firefox_android_release_external_leanplum_eventparameters_v2_20201023\n'
[2020-10-24 04:21:08,012] {logging_mixin.py:112} INFO - [2020-10-24 04:21:08,012] {pod_launcher.py:125} INFO - b'INFO:root:Creating external table tmp.firefox_android_release_external_leanplum_events_v2_20201023\n'
[2020-10-24 04:22:08,176] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,176] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2020-10-24 04:22:08,176] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,176] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 426, in _make_request\n'
[2020-10-24 04:22:08,200] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,200] {pod_launcher.py:125} INFO - b' six.raise_from(e, None)\n'
[2020-10-24 04:22:08,201] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,201] {pod_launcher.py:125} INFO - b' File "<string>", line 3, in raise_from\n'
[2020-10-24 04:22:08,202] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,202] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 421, in _make_request\n'
[2020-10-24 04:22:08,202] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,202] {pod_launcher.py:125} INFO - b' httplib_response = conn.getresponse()\n'
[2020-10-24 04:22:08,202] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,202] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/http/client.py", line 1347, in getresponse\n'
[2020-10-24 04:22:08,238] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,238] {pod_launcher.py:125} INFO - b' response.begin()\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/http/client.py", line 307, in begin\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b' version, status, reason = self._read_status()\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/http/client.py", line 268, in _read_status\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b' line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/socket.py", line 704, in readinto\n'
[2020-10-24 04:22:08,274] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,274] {pod_launcher.py:125} INFO - b' return self._sock.recv_into(b)\n'
[2020-10-24 04:22:08,274] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,274] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/ssl.py", line 1241, in recv_into\n'
[2020-10-24 04:22:08,316] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,315] {pod_launcher.py:125} INFO - b' return self.read(nbytes, buffer)\n'
[2020-10-24 04:22:08,317] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,317] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/ssl.py", line 1099, in read\n'
[2020-10-24 04:22:08,321] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,321] {pod_launcher.py:125} INFO - b' return self._sslobj.read(len, buffer)\n'
[2020-10-24 04:22:08,321] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,321] {pod_launcher.py:125} INFO - b'socket.timeout: The read operation timed out\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,321] {pod_launcher.py:125} INFO - b'\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,322] {pod_launcher.py:125} INFO - b'During handling of the above exception, another exception occurred:\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,322] {pod_launcher.py:125} INFO - b'\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,322] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,322] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send\n'
[2020-10-24 04:22:08,337] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,336] {pod_launcher.py:125} INFO - b' resp = conn.urlopen(\n'
[2020-10-24 04:22:08,337] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,337] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 726, in urlopen\n'
[2020-10-24 04:22:08,341] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,341] {pod_launcher.py:125} INFO - b' retries = retries.increment(\n'
[2020-10-24 04:22:08,342] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,342] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 403, in increment\n'
[2020-10-24 04:22:08,365] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,365] {pod_launcher.py:125} INFO - b' raise six.reraise(type(error), error, _stacktrace)\n'
[2020-10-24 04:22:08,365] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,365] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 735, in reraise\n'
[2020-10-24 04:22:08,374] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,374] {pod_launcher.py:125} INFO - b' raise value\n'
[2020-10-24 04:22:08,374] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,374] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 670, in urlopen\n'
[2020-10-24 04:22:08,375] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,374] {pod_launcher.py:125} INFO - b' httplib_response = self._make_request(\n'
[2020-10-24 04:22:08,375] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,375] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 428, in _make_request\n'
[2020-10-24 04:22:08,375] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,375] {pod_launcher.py:125} INFO - b' self._raise_timeout(err=e, url=url, timeout_value=read_timeout)\n'
[2020-10-24 04:22:08,375] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,375] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 335, in _raise_timeout\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b' raise ReadTimeoutError(\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b"urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=60)\n"
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'During handling of the above exception, another exception occurred:\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2020-10-24 04:22:08,377] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b' File "/usr/local/bin/leanplum-data-export", line 33, in <module>\n'
[2020-10-24 04:22:08,420] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,420] {pod_launcher.py:125} INFO - b" sys.exit(load_entry_point('leanplum-data-export', 'console_scripts', 'leanplum-data-export')())\n"
[2020-10-24 04:22:08,420] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,420] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__\n'
[2020-10-24 04:22:08,432] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,432] {pod_launcher.py:125} INFO - b' return self.main(*args, **kwargs)\n'
[2020-10-24 04:22:08,432] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,432] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main\n'
[2020-10-24 04:22:08,437] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,437] {pod_launcher.py:125} INFO - b' rv = self.invoke(ctx)\n'
[2020-10-24 04:22:08,437] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,437] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b' return _process_result(sub_ctx.command.invoke(sub_ctx))\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b' return ctx.invoke(self.callback, **ctx.params)\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b' return callback(*args, **kwargs)\n'
[2020-10-24 04:22:08,439] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b' File "/app/leanplum_data_export/leanplum_data_export/__main__.py", line 31, in export_leanplum\n'
[2020-10-24 04:22:08,463] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,463] {pod_launcher.py:125} INFO - b' exporter.export(date, s3_bucket, bucket, prefix, bq_dataset, table_prefix, version, clean)\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b' File "/app/leanplum_data_export/leanplum_data_export/export.py", line 62, in export\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b' self.create_external_tables(gcs_bucket, prefix, date, self.DATA_TYPES,\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b' File "/app/leanplum_data_export/leanplum_data_export/export.py", line 208, in create_external_tables\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b' self.bq_client.delete_table(table, not_found_ok=True)\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 1055, in delete_table\n'
[2020-10-24 04:22:08,510] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,510] {pod_launcher.py:125} INFO - b' self._call_api(retry, method="DELETE", path=table.path)\n'
[2020-10-24 04:22:08,512] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,512] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 476, in _call_api\n'
[2020-10-24 04:22:08,512] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,512] {pod_launcher.py:125} INFO - b' return call()\n'
[2020-10-24 04:22:08,512] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,512] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func\n'
[2020-10-24 04:22:08,538] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,538] {pod_launcher.py:125} INFO - b' return retry_target(\n'
[2020-10-24 04:22:08,539] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,539] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 184, in retry_target\n'
[2020-10-24 04:22:08,540] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,540] {pod_launcher.py:125} INFO - b' return target()\n'
[2020-10-24 04:22:08,540] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,540] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 424, in api_request\n'
[2020-10-24 04:22:08,554] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,554] {pod_launcher.py:125} INFO - b' response = self._make_request(\n'
[2020-10-24 04:22:08,554] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,554] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 288, in _make_request\n'
[2020-10-24 04:22:08,556] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,555] {pod_launcher.py:125} INFO - b' return self._do_request(\n'
[2020-10-24 04:22:08,556] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,556] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 326, in _do_request\n'
[2020-10-24 04:22:08,556] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,556] {pod_launcher.py:125} INFO - b' return self.http.request(\n'
[2020-10-24 04:22:08,556] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,556] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/google/auth/transport/requests.py", line 464, in request\n'
[2020-10-24 04:22:08,559] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b' response = super(AuthorizedSession, self).request(\n'
[2020-10-24 04:22:08,559] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 533, in request\n'
[2020-10-24 04:22:08,559] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b' resp = self.send(prep, **send_kwargs)\n'
[2020-10-24 04:22:08,559] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 646, in send\n'
[2020-10-24 04:22:08,560] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b' r = adapter.send(request, **kwargs)\n'
[2020-10-24 04:22:08,560] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,560] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 529, in send\n'
[2020-10-24 04:22:08,560] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,560] {pod_launcher.py:125} INFO - b' raise ReadTimeout(e, request=request)\n'
[2020-10-24 04:22:08,561] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,561] {pod_launcher.py:125} INFO - b"requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=60)\n"
[2020-10-24 04:22:09,647] {logging_mixin.py:112} INFO - [2020-10-24 04:22:09,646] {pod_launcher.py:142} INFO - Event: firefox-android-release-export-af18059c had an event of type Failed
[2020-10-24 04:22:09,647] {logging_mixin.py:112} INFO - [2020-10-24 04:22:09,647] {pod_launcher.py:234} INFO - Event with job id firefox-android-release-export-af18059c Failed
[2020-10-24 04:22:09,687] {logging_mixin.py:112} INFO - [2020-10-24 04:22:09,687] {pod_launcher.py:142} INFO - Event: firefox-android-release-export-af18059c had an event of type Failed
[2020-10-24 04:22:09,687] {logging_mixin.py:112} INFO - [2020-10-24 04:22:09,687] {pod_launcher.py:234} INFO - Event with job id firefox-android-release-export-af18059c Failed