How to remotely control *any* iOS app!

rs2000

I have just filed a bug report to Apple including steps to reproduce the issue.

rs2000

@Janosax said:

@rs2000 said:

@rs2000 said:

@Janosax said:
Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

iOS 11.4.1 definitely works.

This is worth a bug report @ Apple me thinks.

Yes that’s worth it for sure. The issue with Apple bug report is their lack of communication. I’ve reported to them the GarageBand undo bug some months ago, they asked me for some song files and then no news. It has been fixed lately, so that’s cool, but the user need to test from time to time after an update to know it. However, I will look to report this one too.

Just a question: as you assign a recipe to « launch recipe », does that mean only this recipe will works? Or you can have several recipes?

I recommend to give the recipe the same name as your app under control.
One recipe for each app feels like the most logical way to handle different recipes.

Carnbot

Will have a go with a bt keyboard, confirmed to not work for me with a midi device.

rs2000

@aplourde said:

@rs2000 said:

@rs2000 said:

@Janosax said:
Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

iOS 11.4.1 definitely works.

This is worth a bug report @ Apple me thinks.

@rs2000 Thanks for this - another very cool technique to play with!

I'm on iOS 12.1 and I can get this to work with a Bluetooth keyboard, but not a MIDI keyboard. I can assign the Switches, but I can't trigger the Recipes; gonna keep playing with this.

The idea is especially appealing for apps that have gestural controls, but no motion recording (SpaceCraft, Shoom, etc.) The end result isn't as clean as true motion recording as you have to pantomime the gestures on the blank recording screen (I wish you could load a screenshot...) and sometimes apps don't quite respond correctly. Still this is very fun and very cool.

Like I said before, you can use an overhead pen and paint on the iDevice screen, you can remove it easily with a small piece of paper towel dipped into alcohol.
A bit of practice certainly doesn't hurt

For Genome MIDI, I've recorded a set of gestures that each actually does five things:
tap "back" to go back to the session/pattern view
tap "+" to create a new empty pattern on a new track
tap on the new pattern to select it
tap the "stylus" to edit the pattern in the piano roll
tap "REC" to enable recording in the piano roll
This is basically a MIDI looper that can be controlled (6 tracks in my case) by a footswitch.

Genome will loop the pattern while recording and you can add notes the way you want in each cycle (overdub recording).

aplourde

@rs2000 said:

@aplourde said:

@rs2000 said:

@rs2000 said:

@Janosax said:
Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

iOS 11.4.1 definitely works.

This is worth a bug report @ Apple me thinks.

@rs2000 Thanks for this - another very cool technique to play with!

I'm on iOS 12.1 and I can get this to work with a Bluetooth keyboard, but not a MIDI keyboard. I can assign the Switches, but I can't trigger the Recipes; gonna keep playing with this.

The idea is especially appealing for apps that have gestural controls, but no motion recording (SpaceCraft, Shoom, etc.) The end result isn't as clean as true motion recording as you have to pantomime the gestures on the blank recording screen (I wish you could load a screenshot...) and sometimes apps don't quite respond correctly. Still this is very fun and very cool.

Like I said before, you can use an overhead pen and paint on the iDevice screen, you can remove it easily with a small piece of paper towel dipped into alcohol.

I wouldn't go crazy with this! The iPad has an oleophobic coating that will be degraded by alcohol. Occasional use is probably fine, but I wouldn't treat it like an overhead projector in a school!

A bit of practice certainly doesn't hurt

No doubt, but trying to do a musical performance blind and without audio is a challenge!

I just tried playing through a piece of paper and it works. I'm going to try printing out the screen 1:1 and taping that on when recording gestures. This wouldn't work for screen automation where you're going to different screens, but I really want this to work with Shoom (It works, I'm just flying blind and deaf...)

For Genome MIDI, I've recorded a set of gestures that each actually does five things:
tap "back" to go back to the session/pattern view
tap "+" to create a new empty pattern on a new track
tap on the new pattern to select it
tap the "stylus" to edit the pattern in the piano roll
tap "REC" to enable recording in the piano roll
This is basically a MIDI looper that can be controlled (6 tracks in my case) by a footswitch.

Genome will loop the pattern while recording and you can add notes the way you want in each cycle (overdub recording).

Very cool.

Carnbot

I had another look at this, working better for me with a Bluetooth keyboard but still buggy, the gesture recording doesn’t playback the gesture properly in the right place.
I can see the potential for it more though now, the keyboard works for highlighting and selecting focus points, so yeah could be useful if they fixed it.

Hmtx

OK. I'm getting this set up and working but I can't figure out how to stop the scanning? In Loopy HD it keeps scanning sets of four loops, and changing focus with the blue box. It never scans the whole screen.

Hmtx

also, I'm pretty hyped about my standard logitech wireless keyboard working (at all) on my iPad. Sure enough, works great. I thought that only bluetooth keyboards work with iOS so I have never even tried using my full-size keyboard with the iPad.

aplourde

@Hmtx said:
OK. I'm getting this set up and working but I can't figure out how to stop the scanning? In Loopy HD it keeps scanning sets of four loops, and changing focus with the blue box. It never scans the whole screen.

Set "Scanning Style" (below the Switches and Recipes items) to Manual to stop the scanning.

Hmtx

Yeah, scanning style is set to manual. But that only stops the scanning. The focus just stays on the first item in each app.

rs2000

@Hmtx said:
Yeah, scanning style is set to manual. But that only stops the scanning. The focus just stays on the first item in each app.

If you still see scanning frames or lines then something is wrong.
Using a wireless or Bluetooth keyboard definitely allows me to directly tap/toggle Loopy's Play button and each loop circle, even on an iPhone with iOS 12. Loopy looks no different with Switch Control enabled, everything happens "silently".

aplourde

@Hmtx said:
Yeah, scanning style is set to manual. But that only stops the scanning. The focus just stays on the first item in each app.

It should disappear after the Auto Hide time....

Janosax

@Hmtx it seems you have to assign your recipe to « Launch recipe » to avoid that behavior:

Hmtx

@Janosax you go it! Yeah, for some reason I didn't really understand that. So, it has to be set up to "launch recipe" when switch control is enabled. And put all your switches into that recipe. Success!

I can now connect my Logitech keyboard and draw a smiley face in Notes with a press of numpad 3 and a frownie face with a press of numpad 1. Now on to more useful things.

rs2000

@Hmtx said:
@Janosax you go it! Yeah, for some reason I didn't really understand that. So, it has to be set up to "launch recipe" when switch control is enabled. And put all your switches into that recipe. Success!

My text is too long, isn't it?
Congrats and have fun!

Hmtx

@rs2000 said:
My text is too long, isn't it?
Congrats and have fun!

or my attention span is not long enough

thanks for this

Dr_M

Oh crap > @aplourde said:

@rs2000 said:

@rs2000 said:

@Janosax said:
Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

iOS 11.4.1 definitely works.

This is worth a bug report @ Apple me thinks.

@rs2000 Thanks for this - another very cool technique to play with!

I'm on iOS 12.1 and I can get this to work with a Bluetooth keyboard, but not a MIDI keyboard. I can assign the Switches, but I can't trigger the Recipes; gonna keep playing with this.

The idea is especially appealing for apps that have gestural controls, but no motion recording (SpaceCraft, Shoom, etc.) The end result isn't as clean as true motion recording as you have to pantomime the gestures on the blank recording screen (I wish you could load a screenshot...) and sometimes apps don't quite respond correctly. Still this is very fun and very cool.

Oh man, automations triggered by a BT Numeric keypad?! 🤯

rs2000

@Dr_M said:
Oh crap > @aplourde said:

@rs2000 said:

@rs2000 said:

@Janosax said:
Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

iOS 11.4.1 definitely works.

This is worth a bug report @ Apple me thinks.

@rs2000 Thanks for this - another very cool technique to play with!

I'm on iOS 12.1 and I can get this to work with a Bluetooth keyboard, but not a MIDI keyboard. I can assign the Switches, but I can't trigger the Recipes; gonna keep playing with this.

The idea is especially appealing for apps that have gestural controls, but no motion recording (SpaceCraft, Shoom, etc.) The end result isn't as clean as true motion recording as you have to pantomime the gestures on the blank recording screen (I wish you could load a screenshot...) and sometimes apps don't quite respond correctly. Still this is very fun and very cool.

Oh man, automations triggered by a BT Numerical Keyboard?! 🤯

💡👌🏼🌈☀️🍾🥂

Dr_M

@rs2000 THANK YOU FOR BREACHING THIS SUBJECT

rs2000

@Dr_M said:
@rs2000 THANK YOU FOR BREACHING THIS SUBJECT

You're welcome
I just found it too exciting to keep it for myself

Hmtx

this is just mind-boggling. If I have a loopy track that I want to load up in Sector and get glitchy... done with a tap of a button, maybe even mid-song?

ksound

@rs2000 said:

@Hmtx said:
Yeah, scanning style is set to manual. But that only stops the scanning. The focus just stays on the first item in each app.

If you still see scanning frames or lines then something is wrong.
Using a wireless or Bluetooth keyboard definitely allows me to directly tap/toggle Loopy's Play button and each loop circle, even on an iPhone with iOS 12. Loopy looks no different with Switch Control enabled, everything happens "silently".

Thanks so much for documenting this, @rs2000. I tried setting it up for Gadget transport controls before they opened it up to Midi CC's, but never figured out how to get the scanning lines to turn off.

Erayloop

Video pls!

rs2000

I've just created a set of remote control gestures for my navigation system:
Zoom in, zoom out, move up/down/left/right, re-center and follow GPS position (This is what I need on the motorcycle for spontaneous routing decisions).
Now I'm waiting for this $4 BT game controller to arrive (hopefully it will work):

Janosax

@rs2000 this is now fixed and works PERFECTLY on my iPhone 7 Plus iOS 12.3.1!!!!!

This is MAGIC with Gadget and Nanokey Studio, being able to trigger scenes by midi is a game changer for me, Gadget is so fast to create live music with, have to try with BlueBoard too.

Many thanks again for the tutorial 🙏🥂

rs2000

@Janosax said:
@rs2000 this is now fixed and works PERFECTLY on my iPhone 7 Plus iOS 12.3.1!!!!!

This is MAGIC with Gadget and Nanokey Studio, being able to trigger scenes by midi is a game changer for me, Gadget is so fast to create live music with, have to try with BlueBoard too.

Many thanks again for the tutorial 🙏🥂

Merçi beaucoup pour letting me know!
I have received a notification from Apple that my bug report had been merged with another one but no further info about a fix.
Wow, this is great! Being able to virtually tap anywhere on screen by MIDI is the real deal.
Guess I can start practicing to type e-mails on my piano now

[Deleted User]

Amazing finding... but I'm trying with garageband liveloops/playlist buttons and it seems not tapping properly... there are also lots of options on all these config pages (like long tap etc) which makes it a bit confusing..,
Can someone try and if success, explain what else did?
I read carefully the instructions and set 2 keys on my usb keyboard (recognized and triggering the gesture) but tap in the screen is not happening...

Solved: Gesture should be double tap to make trigger live loops.

JUST THANKS

edit2: it needs some work due tap launches double tap stop... any advice to set the proper gesture for just launching or how to differentiate them?

rs2000

@TheDubbyLabby said:
Amazing finding... but I'm trying with garageband liveloops/playlist buttons and it seems not tapping properly... there are also lots of options on all these config pages (like long tap etc) which makes it a bit confusing..,
Can someone try and if success, explain what else did?
I read carefully the instructions and set 2 keys on my usb keyboard (recognized and triggering the gesture) but tap in the screen is not happening...

Solved: Gesture should be double tap to make trigger live loops.

JUST THANKS

edit2: it needs some work due tap launches double tap stop... any advice to set the proper gesture for just launching or how to differentiate them?

You're welcome and thanks for the feedback! 😄
If you need more than one gesture for the same control then you would record separate gestures for separate remote keys/buttons.
Double-tap might work by triggering a single-tap twice, but for tap-hold I suppose you'll have to record a separate gesture for a different key/message. Theoretically we could even use MidiFire to do some sophisticated MIDI processing and send tap/double-tap alternating but I doubt you will be able to select from which MIDI port to receive in Apple's Accessibility Options. I bet it'll always capture all incoming MIDI messages.

[Deleted User]

@rs2000 said:

@TheDubbyLabby said:
Amazing finding... but I'm trying with garageband liveloops/playlist buttons and it seems not tapping properly... there are also lots of options on all these config pages (like long tap etc) which makes it a bit confusing..,
Can someone try and if success, explain what else did?
I read carefully the instructions and set 2 keys on my usb keyboard (recognized and triggering the gesture) but tap in the screen is not happening...

Solved: Gesture should be double tap to make trigger live loops.

JUST THANKS

edit2: it needs some work due tap launches double tap stop... any advice to set the proper gesture for just launching or how to differentiate them?

You're welcome and thanks for the feedback! 😄
If you need more than one gesture for the same control then you would record separate gestures for separate remote keys/buttons.
Double-tap might work by triggering a single-tap twice, but for tap-hold I suppose you'll have to record a separate gesture for a different key/message. Theoretically we could even use MidiFire to do some sophisticated MIDI processing and send tap/double-tap alternating but I doubt you will be able to select from which MIDI port to receive in Apple's Accessibility Options. I bet it'll always capture all incoming MIDI messages.

For the moment I just need to make work the single tap properly... I haven't gadget so IDK how it differs in behaviour (it differs?) so I'm not sure about doing the whole process right...

Can someone ( @Janosax ?) using it for clip launching try the same for Garageband, please?
I will appreciate it...

mbncp

Is there a way to have your external keyboard(HID) to work normaly when switch control is on ?