Jake
03/03/2022, 2:43 PMKeith Veleba
03/03/2022, 3:15 PMXavier Babu
03/03/2022, 3:52 PMDominick Olivito
03/03/2022, 3:54 PM/home/flex/.local/bin/prefect: line 3: import: command not found
/home/flex/.local/bin/prefect: line 4: import: command not found
/home/flex/.local/bin/prefect: line 5: from: command not found
/home/flex/.local/bin/prefect: prefect: line 7: syntax error near unexpected token `('
/home/flex/.local/bin/prefect: prefect: line 7: ` sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'
it looks like it's finding and parsing the prefect
executable file but not running it with python
. when I run a local container using the image, i'm able to successfully call the command prefect
and run a flow. prefect
is in the path of active user (flex
).
i'm also able to run basic flows successfully on GKE using prefect's base image, so the issue is specific to our custom image. do you have any suggestions on what we can check in our custom image?Vadym Dytyniak
03/03/2022, 4:50 PMChristian Nuss
03/03/2022, 4:51 PMKubernetesRun
defininig the job_template
as a dict?Prasanth Kothuri
03/03/2022, 5:15 PM# upload to s3
write_to_s3 = S3Upload(
bucket=s3_bucket,
boto_kwargs=dict(
endpoint_url=os.getenv("s3_endpoint"),
aws_access_key_id=os.getenv("s3_access_key"),
aws_secret_access_key=os.getenv("s3_secret_key")
)
)
output = write_to_s3(results.to_csv(index=False), key=file_name)
Aram Panasenco
03/03/2022, 6:16 PMIfeanyi Okwuchi
03/03/2022, 8:03 PMcreate_flow_run.map()
with wait_for_flow_run()
and get_task_run_result()
but I'm getting some errors. I'm not sure if I'm doing this correctlySeth Just
03/03/2022, 8:15 PMECSRun
with a LocalDaskExecutor
and I'm unable to find any way to debug or diagnose why the task isn't ever executed.Sam Werbalowsky
03/03/2022, 9:40 PMparam=Parameter("myvalue", default="myvalue")
. I want to us it in a variety of tasks say…
execute = execute_file(f"sql/{param}.sql")
upload = upload_file(f"{param}.csv")
Is there a way to do this without constructing a task for each input?YD
03/03/2022, 9:53 PMtask_1
return a df and I pass the df as an input to task_2
, does the df
physically go to the Prefect server ?
the issue I have is that it looks like it is getting stuck when trying to return the df and pass it to the next task.
is there a better way to pass df from one task to the next ?
@task()
def task_1():
df = ....
return df
@task()
def task_2(df):
df2 = df ....
return df2
def main():
with Flow("voc_sentiment") as flow:
df = task_1()
df2 = task_2(df)
Daniel Saxton
03/04/2022, 12:50 AM--env
flag, but is there also something like an --env-file
flag if you have several environment variables defined in a file (kind of like you have with docker-compose)?
https://docs.prefect.io/orchestration/getting-started/flow-configs.html#configure-environment-variablesMatt Alhonte
03/04/2022, 1:22 AMset_schedule_inactive
). It lets me read (I grab all the flows from a given project), but the mutate
gives me this error (the query works when I go to the Interactive tab on the GUI btw)
AuthorizationError([{'path': ['flow'], 'message': 'AuthenticationError: Forbidden', 'extensions': {'code': 'UNAUTHENTICATED'}}])
Sen
03/04/2022, 5:30 AMhttps://pasteboard.co/kpsNsqpN3aXP.png▾
Thomas Opsomer
03/04/2022, 4:22 PMkevin
03/04/2022, 5:14 PMLing Chen
03/04/2022, 5:46 PMKen Nguyen
03/04/2022, 6:15 PMFROM ubuntu:18.04
COPY requirements.txt /requirements.txt
COPY google_secret.json $HOME/.config/gspread_pandas/google_secret.json
ENV PATH="/root/miniconda3/bin:$PATH"
ARG PATH="/root/miniconda3/bin:$PATH"
RUN apt-get update
RUN apt-get install -y wget && rm -rf /var/lib/apt/lists/*
RUN wget \
<https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh> \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh
RUN conda install pip
RUN pip install -r requirements.txt
RUN conda install -c conda-forge theano-pymc -y
Ken Nguyen
03/04/2022, 6:39 PM@task
decorator on a function imported from a library (as opposed to one I defined)?Jack Chang
03/04/2022, 6:40 PMTypeError: Task is not iterable. If your task returns multiple results, pass `nout` to the task decorator/constructor, or provide a `Tuple` return-type annotation to your task.
Can anyone point me to an example where I can make it iterable?Hedgar
03/04/2022, 6:59 PMDaniel Saxton
03/04/2022, 7:06 PMLeon Kozlowski
03/04/2022, 8:40 PMHenry
03/04/2022, 8:55 PMinput_a = Parameter('a')
@task
def abc(b, c):
return a + b + c
with Flow("abc_flow") as flow:
# in context
result = abc()
Jared Teerlink
03/04/2022, 9:21 PMMartha Edwards
03/04/2022, 9:58 PMKevin Kho
03/04/2022, 11:19 PMKevin Kho
03/04/2022, 11:30 PMHans Lellelid
03/05/2022, 8:39 PMHans Lellelid
03/05/2022, 8:39 PMKevin Kho
03/05/2022, 8:57 PMHans Lellelid
03/05/2022, 9:04 PMKevin Kho
03/05/2022, 9:10 PMMichael Adkins
03/05/2022, 10:36 PMOrionClient.create_flow_run_from_deployment
method which allows you to pass parameters https://orion-docs.prefect.io/api-ref/prefect/client/#prefect.client.OrionClient.create_flow_run_from_deploymentKevin Kho
03/05/2022, 10:38 PMMichael Adkins
03/05/2022, 10:39 PMprefect deployment run
) because parsing parameters from the CLI into JSON is a bit of a pain and I haven’t implemented it, but we will definitely have it in the future.Hans Lellelid
03/07/2022, 2:27 AMKevin Kho
03/07/2022, 2:43 AM