Support recording Desktop Apps

Voters 63

S
V
J
+ 57 voters
A
A
R
A
P
T
Z
R
A
M
A
Y
Z
L
L
W
A
M
V
C
J
A
J
A
T
P
B
A
S
A
R
A
Z
A
J
A
K
R
M
T
J
A
M
A
J

In progress
Subscribe

Kyle

Support recording Desktop Apps so that automations can accomplish more.

01-01-2024 -

Activity Newest / Oldest

Kyle

OCR for desktop automation + Misc Desktop recording improvements are in development.

The automation is proving to work in minimal cases without OCR and screen changes are causing the automations to fail to frequently.

Updates to the installation script will be released in version 1.3.4 - if you are seeing any issues with installation of the desktop plugin please post screenshots in this ticket: request.taskmagic.com/b/bugs/install-of-desktop-controlbeta/

P.S. Most desktop automations can be accomplished with web/apps

1. Open excel document
2. Select sheet
3. Read last row
4. Add new row with information based on last row
5. Save document
6. Close excel

Paul

Articulate Storyline/Rise

zshon

1. Open a Zoom meeting link and maximize the window.
2. Then, launch OBS (Open Broadcaster Software).
3. Once OBS is open, start recording the session.

Jeremy

Replying to this request I just got via email: If anyone can comment ideas below of automations they want to explore with desktop recordings so that myself and other developers can test it as we build it would be helpful! First up is testing automating / perfecting iMessage :)

I don't have a specific use-case. Generally, I'd like to be able to execute any program in my Applications folder, instruct TaskMagic where to click, and then populate with/and or extract variables/data, for example, from/to a google sheet or other online source.

I use locally-installed research tools, that have no web-frontend - although they pull in data and functionality via their own API's etc. Example: Screaming Frog SEOspider - a) open it, b) populate it with a specific URL, d) load specific presets via the applications file-menu, e) grab and store certain result information, f) check completeness/loop thru next, g) exit.

That kind of thing :)

Jeremy

whoop whoop!

Kyle

Status changed to: In progress

Jeremy

this will be awesome!

Robert

Mac Automator please...

Jeremy

Mike

Yes please. I can imagine all the Zimmwriter hours I'll get back!

Marcel Dinescu

I know what you mean! Please do a video explaining the process

SB

This would be fantastic, and if you can have it upload/download/save files as part of automations as well that would be great.

Kyle

If anyone can comment ideas below of automations they want to explore with desktop recordings so that myself and other developers can test it as we build it would be helpful!

First up is testing automating / perfecting iMessage :)

Pascal Noyel

Automating with Mac Shortcuts (Automator) would be great

AppSumo

+1 on IMessage

WhatsApp desktop

Outlook

File Folder tasks (windows/mac) such as bulk moving by parameter or file, bulk renaming (set nomeclature and let it auto-add series), etc

Integration with a photo tool to do bulk shrink

Auto download from browser email and integration with a locally run LLM to perform OCR tasks like text extraction

Nick

The more we can have it just be able to operate like it does in browser (freely), the better.

#1 AI Team in Slack:
- Set up a Slack Automation where specific channels represent specific "AI Team Members".
- Set up Automation --> When message sent to channel - extract message, add to conversation history spreadsheet, with date and time.
- Message is sent to Desktop Automation --> PromptGeneratorAI goes to ChatGPT, uploads conversation history as context, considers my recent message, decides how to respond, adds it to the spreadsheet conversation history, and responds to the message I sent in a threaded reply. All follow up messages are continually updated as if we are having ongoing conversations in realtime.
- Triggered Automation: Every 24 hours, take the conversation history for the day, and run through series of prompt automations in PromptGeneratorAI to create a "Daily Journal" for each "AI Team Member". Allowing them to grow, learn, develop deeper context and nuanced understanding, etc.

1st Team Member Goal: Personal Assistant
After that, I have a ton of specialized assistants I will build out.
_______

OVERALL & for SEO / DIgital PR / Link Building
For me - the desktop automation is all about me being able to create automations that allow my team to access the desktop tools I have access to, directly in a spreadsheet OR the workflows that involve desktop tools that I have no longer require me to do the manual data uploading / downloading / merging:
1) Spreadsheet input --> Data sent to desktop workflow
2) That data and resulting data will be used in a number of ways for SEO / trends analysis / digital PR / etc. usecases with tools like SEOUtils.app, screamingfrog, URL Profiler, PromptGeneratorAI, etc.
- Downloading files / extracting data + combining all that data together in different ways + re-using it with the tools above
3) Data gets output to spreadsheets for team use

Stephan

Since we are already using a windows desktop version, it would be great to also allow for explorer recordings. E.g., open specific folders or files within a workflow.

Kyle

Merged with: Allow Explorer/Desktop recordings

Kyle

Status changed to: Planned

David Mead

Yes please
The whole point of a desktop app

Jay

I would like to see this work with any software/app online and offline (that's already installed on the Device).
MS Office, Corel, Libre Office, Photoshop, Visual Studio etc.