In Amazon-ec2, the instances page shows details of a machine like its IP, size, key-pair, security group, how long it has run etc.
once the instance is terminated, the line-item stays visible for about an hour. within this period, we can know the details of the machine as it was while running. but once the line item gets removed, there is no way to know that.
say, some instances are manually instantiated, used for some time and then terminated. after an hour of that event there is no way to find out what happened.
there is one detailed-bill feature, but it only provides the instance-ids and size. i am interested in key-pair, ip, OS, security group and name-of-machine if any. is there any way to find out them?
Edit
I understand that i can have a cron job periodically list all instances (and its details) and store it in a database. thing is, to host that cron process, i would need a machine 24x7. what i need is sort of hook, a callback, event.
even if not readily available, can such a solution be made?
Once the instance has been terminated, like you mentioned, most of the information will be available through the API before it completely disappears after an hour or so. (IP address an DNS will not be available since every time you stop or terminate an instance the IP address is relinquished) After the instance completely disappears it means that it's gone for good.
The workaround is to query the instances API every so ofter and save the state and instance information. You can save it in memory, a database or just text files, depending on what you are trying to do or what application you are trying to create.
Here's an example of saving the instance information into a Python dictionary in memory using the boto Python interface to the API:
reservations = conn.get_all_instances()
for res in reservations:
instance = res.instances[0]
if instance.id == 'i-xxxxxx':
instance_dict[instance.id] = instance
The dictionary instance_dict will always have the IP address, DNS and other instance info for the duration of your program as long as you don't overwrite it. To terminate the instance you can run something like:
instance_dict['i-xxxxxx'].terminate()
but later you can still use:
instance_dict['i-xxxxxx'].ip_address
and:
instance_dict['i-xxxxxx'].dns_name
Hope this helps.
Related
All,
Am really stuck and have tried almost everything. Can some one please help.
I provision 2 instances while creating my Auto-scaling group. I trigger a Lambda ( manipulates the tags) which changes the instance name to a unique name.
Desired State
I want first instance of Lambda to give first instance the name "web-1"
Then second instance of lambda would run just fine to assign a name "web-2"
Current State
I start with a search on running instances to see if "web-1" exists or not.
So in this case my Lambda executes twice and creates both instances with the same name ( web-1, web-1).
How do I get around this ? I know that the problem is due to Lambda listening to Cloud Watch events. ASG Launch creates 2 events at the same time in my case leading to the problem I have.
Thanks.
You are running into a classic multi-threading issue. Both lambda functions execute simultaneously, see the same "unused" web-1 and mark both with the same function.
What you need is an atomic operation that gives each Lambda execution "permission" to proceed. You can try using a helper DynamoDB table to serialize the tag attempts.
Have your lambda function decide which tag to set (web-1, web-2, etc.)
Check a DynamoDB table to see if that tag has been set in the last 30 seconds. If so, someone else got to it first, so go back to step 1.
Try to write your "ownership" of your sought-after tag to the DynamoDB along with your current timestamp. Try using some attribute_not_exists or other DynamoDB conditions to ensure only one simultaneous such write succeeds.
If you fail at writing, go back to step 1.
If you succeed at writing, then you're free to set your tag.
The reason for the timestamps is to allow for "web-1" to be terminated, and then having a new EC2 instance launched and labelled "web-1".
The above logic is not proven to work, but hopefully should give enough guidance to develop a working solution.
I'm planning to start using Amazon EC2, and, as everyone, I want to use Spot instances.
Will be for a minigames server, so Spot instances are perfect for this. Players enter, play the match and leave, so when a Spot instance finishes because of spot instance price volatility only current match will be finished, barely any data loss and perfectly acceptable when you save a lot of money.
Now, altough players are going to be disconnected and connected to an ondemand server when volatility reaches maximum bid, I would like to know if when a Spot instance is force-terminated is called the normal shutdown command or simply is "unplugged" and I don't have a chance to disconnect players safely and save their data to the database (this will take just a few milliseconds).
As of 2015, Amazon now provides a 2-minute termination notice in the instance metadata.
A custom script can be written to poll for the termination notice and call web server graceful shutdown and associated cleanup scripts to ensure zero impact to end-users.
Source: https://aws.amazon.com/blogs/aws/new-ec2-spot-instance-termination-notices/
I am trying to establish a barrier between to different processes in Windows. They are essentially two copies of the same process (Running them as two separate threads instead of processes is not an option).
The idea is to place barriers at different stages of the program, to make sure that both processes start each stage at the same time.
What is the most efficient way of implementing this in Windows?
Use a named event (see CreateEvent and WaitForSingleObject API functions). You would need two events per barrier - each event created in another instance of the application. Then both instances wait for each other's event. Of course, these events can be reused later for another barrier.
There exists one complexity, though - as event names are globally unique (let's say so for simplicity), each event would have a different name, maybe prefixed by the instance's process ID. So each instance of the application would have to get another instance's ID in order to find the name of the event created by another instance.
If you have a windowed application, you can broadcast a message which will inform the second instance of the application about an existence of the first instance.
I have two EC2 instances. I want that if one finish a job, it will sign the other one to do other stuff.
So, how to make the communication? I don't want to use CURL.. coz it seems like expensive. I think AWS should have some simple solution but I still can't find relevant help in the documentation.
:(
also, how to send data between two instances without giong through SSH in a fast way? I know ssh can be done. but it seems slow. once again, any tool that EC2 provide to do that?
Actually, I need two methods:
1) Instance A tells Instance B to grab the data from Instance A.
This is answered by Adrian that I can use SQS. I will try that.
2) Once Instance B get the signal, then the data (EBS) data in Instance A needs to transfer to Instance B. The amount of data can be big even I zip it. It is around 50 MB. And I need Instance B to get the data fast so that Instance B will have enough time to process the data before next interval comes in.
So, I am thinking of either these methods:
a) Instance A has the data dump from DB, upload to S3. Then signal Instance B. Instance B gets the data from S3.
b) Instance A has the data dump from DB. Then signal Instance B. Instance B establish SSH (or any connection) to Instance A and grabs the data.
The data may need to be stored permanently but it is not a concern at this moment. It is mainly for Instance B to process.
This is a simple scenario. I'm thinking of what if I scale it with multiple instances, what the proper approach is. :)
Thanks.
Amazon has a special service for this -- it's called SQS, and it allows instances to send messages to each other through special queues. There are SDKs for SQS in various languages, like Java and PHP. This should serve your signaling needs.
For actually sending the bulky data over, it's best to use S3 (and send the object key in the SQS message). You're right that you're introducing latency by adding the extra middle-man, but you'll find that S3 is very fast from EC2 instances (if you put them in the same availability zone, that is), and more importantly than performance, S3 is very reliable. If you try to manage the transfer yourself through SSH, you'll have to work out a lot of error checking and retry logic that S3 handles for you. You can use S3FS to easily write and read to/from S3 from EC2.
Edited to address your updated question.
You may want to look at SNS... which is kind of like push SQS.
How fast do you need this communication to be? SSH is pretty darn fast. The only thing that I can think of that might be faster is raw sockets (from within whatever program is running the jobs).
You could use a distributed workflow managing service.
If Instance B has already completed the task, it can go on to pick another task. Usually, you would want Instance B to signal that is has "picked" up a task and is doing it. Then other instances should try to pick up other tasks on your list. You need a central service which knows which task has been picked up already, and which ones are left for grabs.
When Instance B completes the task successfully, it should signal the central service that it is free for a new task, and pick one up if there is something left.
If it fails to complete the task, the central service should be able to detect it (via heartbeats and timeouts you defined) and put the task back on the list so that some other instance can pick it up.
Amazon SWF is the central service which will provide you with all of this.
For data required by each instance, you should put it in a central store like s3, and configure s3 paths in a way such that each task knows where to download data from, without having to sync up.
e.g. data for task 1 could be placed in something like s3://my-bucket/task1
I have a certain resouce I want to limit access to. Basically, I am using a session level lock. However, it is getting to be a pain writing JavaScript that covers every possible way a window can close.
Once the user leaves that page I would like to unlock the resouce.
My basic idea is to use some sort of server side timeout, to unlock the resouce. Basically, if I fail to unlock the resource, I want a timer to kick in and unlock the resouce.
For example, after 30 seconds with now update from the clientside, unlock the resouce.
My basic question, is what sort of side trick can I use to do this? It is my understanding, that I can't just create a thread in JSF, because it would be unmanaged.
I am sure other people do this kind of thing, what is the correct thing to use?
Thanks,
Grae
As BalusC right fully asked, the big question is at what level of granularity would you like to do this locking? Per logged-in user, for all users, or perhaps you could get away with locking per request?
Or, and this will be a tougher one, is the idea that a single page request grabs the lock and then that specific page is intended to keep the lock between requests? E.g. as a kind of reservation. I'm browsing a hotel page, and when I merely look at a room I have made an implicit reservation in the system for that room so it can't happen that somebody else reserves the room for real while I'm looking at it?
In the latter case, maybe the following scheme would work:
In application scope, define a global concurrent map.
Keys of the map represent the resources you want to protect.
Values of the map are a custom structure which hold a read write lock (e.g. ReentrantReadWriteLock), a token and a timestamp.
In application scope, there also is a single global lock (e.g. ReentrantLock)
Code in a request first grabs the global lock, and quickly checks if the entry in the map is there.
If the entry is there it is taken, otherwise it's created. Creation time should be very short. The global lock is quickly released.
If the entry was new, it's locked via its write lock and a new token and timestamp are created.
If the entry was not new, it's locked via its read lock
if the code has the same token, it can go ahead and access the protected resource, otherwise it checks the timestamp.
If the timestamp has expired, it tries to grab the write lock.
The write lock has a time-out. When the time-out occurs give up and communicate something to the client. Otherwise a new token and timestamp are created.
This just the general idea. In a Java EE application that I have build I have used something similar (though not exactly the same) and it worked quite well.
Alternatively you could use a quartz job anyway that periodically removed the stale entries. Yet another alternative for that is replacing the global concurrent map with e.g. a JBoss Cache or Infinispan instance. These allow you to define an eviction policy for their entries, which saves you from having to code this yourself. If you have never used those caches though, learning how to set them up and configuring them correctly can be more trouble than just building a simple quartz job yourself.